Kubernetes - serviceaccount authentication tokens

When pods communicate with the API server, they use a serviceaccount to authenticate. 

The token for this is defined in a secret, which is mounted as a volume into the pod in question.

The token is signed by the key specified by the API servers --service-account-key-file property

You can test this token my making a request to the API server with curl as follows:

curl -H  “Authorization: Bearer <token>"  https://<master ip addr>:6443/api/v1/namespaces/<namespace>/pods/<pod name>

You can derive the token from the relevant secret as follows:

kubectl describe  secrets <secret name> -n kube-system

The result will look similar to below:

Name:         default-token-jvrkf
Namespace:    kube-system
Labels:       
Annotations:  kubernetes.io/service-account.name=default
              kubernetes.io/service-account.uid=ba8add2d-12ab-11e8-9719-080027dd95a3

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1066 bytes
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkZWZhdWx0LXRva2VuLWp2cmtmIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImRlZmF1bHQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJiYThhZGQyZC0xMmFiLTExZTgtOTcxOS0wODAwMjdkZDk1YTMiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06ZGVmYXVsdCJ9.JYVNgnSNO-Gs4zuqjZHOQs1e4RG31c_trE5AcJXqlexM5On0SriWjb-qICS2o5_M0jNmSKED67lfAomoCsKOk7a4zlAdpkc4eVWxMXEaWemu6yxQAIcJjiaBgEpOm38FMQJKEsb1D3kdD0W0HD1MNcgY0LXxyXuWv7NToqGxbYn7QJqrEEFTM5qEx5IAAA7YeWOkXqQ35Vtg1Co1u-Ilw2Cr1GPtmCazCL0t8bpP4oi7qDr98lYmWNMo8KcPZpIBoS9Cg0bX3gONOE3Fga6Aa0e6VxLPy9VyXdezXvNtsmMtW-ryJ8fvplap2wj2JPFWCEO26DIXQSx8OX_h3ds_ug

Here's an example curl request:

curl -k -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkZWZhdWx0LXRva2VuLWp2cmtmIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImRlZmF1bHQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJiYThhZGQyZC0xMmFiLTExZTgtOTcxOS0wODAwMjdkZDk1YTMiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06ZGVmYXVsdCJ9.JYVNgnSNO-Gs4zuqjZHOQs1e4RG31c_trE5AcJXqlexM5On0SriWjb-qICS2o5_M0jNmSKED67lfAomoCsKOk7a4zlAdpkc4eVWxMXEaWemu6yxQAIcJjiaBgEpOm38FMQJKEsb1D3kdD0W0HD1MNcgY0LXxyXuWv7NToqGxbYn7QJqrEEFTM5qEx5IAAA7YeWOkXqQ35Vtg1Co1u-Ilw2Cr1GPtmCazCL0t8bpP4oi7qDr98lYmWNMo8KcPZpIBoS9Cg0bX3gONOE3Fga6Aa0e6VxLPy9VyXdezXvNtsmMtW-ryJ8fvplap2wj2JPFWCEO26DIXQSx8OX_h3ds_ug" https://192.168.99.100:8443/api/v1/namespaces/kube-system/pods/kube-dns-54cccfbdf8-xqgz6 

You can see how the pod uses this token by running kubectl describe pods kube-dns-54cccfbdf8-xqgz6 -n kube-system and checking the Mounts section in the output.

Mounts:
  /kube-dns-config from kube-dns-config (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from default-token-jvrkf (ro)

It's worth noting that if you run kubectl get secrets <secret name> -n kube-system -o yaml as opposed to kubectl describe secrets <secret name> -n kube-system, the token will be base64 encoded so you'll need to pipe it through base64 --decode before you copy and paste it into your curl command

If you regenerate your cluster certificates for any reason, you'll need to regenerate these tokens. To do this just delete the secrets with kubectl, new secrets will be generated automatically. However (and this is really important and had me going for a while)  these secrets will have a different name, therefore your pods will still be trying to mount the old secrets. To resolve this you also need to delete the pods (e.g. kubectl delete pod <pod1 name> <pod2 name> <pod3 name>) -n kube-system . If the pods are managed by deployments (and this is why you should use deployments by the way) they will automatically be regenerated and will mount the new secrets automatically. 

Your best friend by far in all of this is the command journalctl -u kubelet -f. If you have a pod stuck in ContainerCreating and therefore are not able to look at that it's container logs, simply log on the node in question and run journalctl -u kubelet -f | grep <pod name> .. priceless tool 

NOTE: if you're using Calico networking, this new token will need to be set in /etc/cni/net.d/10-calico.conf which is in turn derived from the ConfigMap calico-config. If you edit the contents of this file directly, the changes will be overwritten by the contents of the calico-config ConfigMap at the next kubelet restart.