Solving access problems after GKE cluster upgrade to v1.3

In this post I’m just going to briefly describe a work-around to a problem I encountered when upgrading our Kubernetes cluster on Google Container Engine (GKE).

The Problem

The problem occurred after upgrading from Kubernetes version 1.2.5 to 1.3.5 on Google Container Engine. After this upgrade I could only perform read operations on the cluster with my user account. For example I could list all the pods just fine by doing:

$ kubectl get pods

but as soon I wanted to do something else like deleting a pod or replication controller the following error was shown:

Error from server: the server does not allow access to the requested resource (...)

Usually one simply calls

gcloud container clusters get-credentials 

to get the credentials but this didn’t make any difference. After quite a bit of research I turned to the GKE Slack channel (#google-containers) and luckily Jeff Hodges (@jmhodges) pointed me in the right direction. It turns out that starting in Kubernetes v1.3 GKE users can authenticate to the Kubernetes API on their cluster using Google OAuth2 access tokens. But something is/was broken on the GKE when upgrading the cluster which meant that I could no longer authenticate correctly.

The Solution

The documentation indicates that you can revert to using the legacy cluster certificate or username/password that you used in the previous version to authenticate. This turns out to be the work-around I was looking for. What one should do is to run these two commands:

$ gcloud config set container/use_client_certificate True
$ export CLOUDSDK_CONTAINER_USE_CLIENT_CERTIFICATE=True

Afterwards make sure to get the credentials again:

gcloud container clusters get-credentials 

Now you should be able to delete pods again! To make this setting permanent you should add “`export CLOUDSDK_CONTAINER_USE_CLIENT_CERTIFICATE=True`” to your `.bashrc` or `.bash_profile`.

55 thoughts on “Solving access problems after GKE cluster upgrade to v1.3

  1. I couldn’t not read/write to the server via kubectl, and resetting that prop made it work. Thakn you!

  2. I don’t think the title of your article matches the content lol. Just kidding, mainly because I had some doubts after reading the article.

  3. I don’t think the title of your article matches the content lol. Just kidding, mainly because I had some doubts after reading the article.

  4. Thank you for your sharing. I am worried that I lack creative ideas. It is your article that makes me full of hope. Thank you. But, I have a question, can you help me?

  5. I am extremely impressed along with your writing talents and also with the layout in your blog. Is this a paid theme or did you customize it your self? Either way keep up the nice high quality writing, it’s rare to see a great blog like this one these days!

  6. I am extremely inspired with your writing talents as well as with the structure in your weblog. Is this a paid subject matter or did you customize it yourself? Anyway stay up the excellent quality writing, it’s rare to peer a nice weblog like this one these days!

  7. I’m really inspired with your writing abilities as well as
    with the format in your weblog. Is that this a
    paid topic or did you customize it your self? Anyway stay up the excellent quality writing, it is rare to peer a nice blog like this one these days.

    Fiverr Affiliate!

Leave a Reply

Your email address will not be published. Required fields are marked *