If you have a pod in ContainerCreating state, follow these steps to held determine the issue:
- carry out a wide pod listing (kubectl get pods -o wide) to determine which node your erroneous pod is hosted on
- ssh to the node in question
- run (as root) journalctl --unit kubelet --no-pager | grep <your pod name>
- In this particular case the following message was present
Mar 15 12:35:49 jupiter.stack1.com kubelet[3146]: E0315 12:35:49.215454 3146 pod_workers.go:186] Error syncing pod ec706dba-2847-11e8-b8d8-0676d5f18210 ("kube-proxy-nwp6t_kube-system(ec706dba-2847-11e8-b8d8-0676d5f18210)"), skipping: failed to "CreatePodSandbox" for "kube-proxy-nwp6t_kube-system(ec706dba-2847-11e8-b8d8-0676d5f18210)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-proxy-nwp6t_kube-system(ec706dba-2847-11e8-b8d8-0676d5f18210)\" failed: rpc error: code = Unknown desc = failed to create a sandbox for pod \"kube-proxy-nwp6t\": Error response from daemon: devmapper: Thin Pool has 152128 free data blocks which is less than minimum required 163840 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior"
- This is stating that the Docker device mapper is out of space (as set in /etc/sysconfig/docker via option --storage-opt dm.loopdatasize=40G) .
- This can usually be resolved by removing containers that have exited, dangling images and dangling volumes using the following steps:
- Cleanup exited processes:
docker rm $(docker ps -q -f status=exited)
- Cleanup dangling volumes:
docker volume rm $(docker volume ls -qf dangling=true)
- Cleanup dangling images:
docker rmi $(docker images --filter "dangling=true" -q --no-trunc)