In this post I’m just going to briefly describe a work-around to a problem I encountered when upgrading our Kubernetes cluster on Google Container Engine (GKE).

The Problem

The problem occurred after upgrading from Kubernetes version 1.2.5 to 1.3.5 on Google Container Engine. After this upgrade I could only perform read operations on the cluster with my user account. For example I could list all the pods just fine by doing:

$ kubectl get pods

but as soon I wanted to do something else like deleting a pod or replication controller the following error was shown:

Error from server: the server does not allow access to the requested resource (...)

Usually one simply calls

gcloud container clusters get-credentials 

to get the credentials but this didn’t make any difference. After quite a bit of research I turned to the GKE Slack channel (#google-containers) and luckily Jeff Hodges (@jmhodges) pointed me in the right direction. It turns out that starting in Kubernetes v1.3 GKE users can authenticate to the Kubernetes API on their cluster using Google OAuth2 access tokens. But something is/was broken on the GKE when upgrading the cluster which meant that I could no longer authenticate correctly.

The Solution

The documentation indicates that you can revert to using the legacy cluster certificate or username/password that you used in the previous version to authenticate. This turns out to be the work-around I was looking for. What one should do is to run these two commands:

$ gcloud config set container/use_client_certificate True
$ export CLOUDSDK_CONTAINER_USE_CLIENT_CERTIFICATE=True

Afterwards make sure to get the credentials again:

gcloud container clusters get-credentials 

Now you should be able to delete pods again! To make this setting permanent you should add “`export CLOUDSDK_CONTAINER_USE_CLIENT_CERTIFICATE=True`” to your `.bashrc` or `.bash_profile`.