Where are my container images when running kubernetes with containerd?

Question

This output says that I'm running kubernetes with containerd as the container runtime:

k get nodes -o wide
NAME          STATUS   ROLES                  AGE   VERSION        INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
k8s-worker3   Ready    <none>                 12d   v1.24.4+k3s1   10.16.24.123   <none>        Ubuntu 20.04.2 LTS   5.15.0-48-generic   containerd://1.6.6-k3s1
k8s-worker1   Ready    <none>                 12d   v1.24.4+k3s1   10.16.24.121   <none>        Ubuntu 20.04.2 LTS   5.13.0-44-generic   containerd://1.6.6-k3s1
k8s-master    Ready    control-plane,master   12d   v1.24.4+k3s1   10.16.24.120   <none>        Ubuntu 20.04.4 LTS   5.15.0-46-generic   containerd://1.6.6-k3s1
k8s-worker2   Ready    <none>                 12d   v1.24.4+k3s1   10.16.24.122   <none>        Ubuntu 20.04.2 LTS   5.13.0-44-generic   containerd://1.6.6-k3s1

I'm deploying one of my pods, it gets scheduled on node k8s-worker3, and kubectl describe pods/mypod says the image was already on the node.

But when I run ctr on the node it shows that there NO images:

user@k8s-worker3:~$ sudo ctr images list
REF TYPE DIGEST SIZE PLATFORMS LABELS

And docker images doesn't show the correct version of the image.

Here's the processes running containerd:

user@k8s-worker3:~$ ps -ef | grep container
root         985       1  0 15:23 ?        00:00:00 /usr/bin/containerd
root        1106       1  0 15:23 ?        00:00:01 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root        1312    1057  0 15:23 ?        00:00:12 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd
root        1918       1  0 15:23 ?        00:00:03 /var/lib/rancher/k3s/data/577968fa3d58539cc4265245941b7be688833e6bf5ad7869fa2afe02f15f1cd2/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5abd3c3104abf812422757d9534c33475819cbf43b64845dd23f535193fed09e -address /run/k3s/containerd/containerd.sock
root        1970       1  0 15:23 ?        00:00:00 /var/lib/rancher/k3s/data/577968fa3d58539cc4265245941b7be688833e6bf5ad7869fa2afe02f15f1cd2/bin/containerd-shim-runc-v2 -namespace k8s.io -id 8c339c15ab8abb3050810b5fcaed817363fcf3b06800b554711b1fa0e95006e2 -address /run/k3s/containerd/containerd.sock
root        3231       1  0 15:25 ?        00:00:01 /var/lib/rancher/k3s/data/577968fa3d58539cc4265245941b7be688833e6bf5ad7869fa2afe02f15f1cd2/bin/containerd-shim-runc-v2 -namespace k8s.io -id 354aa1df3c2a8492a4842efe3882785553501d77f61d9b5ef0bf3343ace2a518 -address /run/k3s/containerd/containerd.sock
user      4957    3094  0 15:48 pts/0    00:00:00 grep --color=auto container

So what is happening here? Where is the image on the node??

Answer 1

I think that crictl is the best solution to monitor images and containers on the containerd (K8s).

Crictl details and commands:

https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md 

Install crictl with curl:

VERSION="v1.26.0" # check latest version in /releases page
curl -L https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-${VERSION}-linux-amd64.tar.gz --output crictl-${VERSION}-linux-amd64.tar.gz
sudo tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin
rm -f crictl-$VERSION-linux-amd64.tar.gz

Configure to run crictl:

sudo nano /etc/crictl.yaml

# copy followings: 
runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10
debug: true

Sample commands: List pods:

sudo crictl pods

List images:

sudo crictl images

List all pods, containers, images, find the containerid:

sudo crictl ps -a

Logs for specific container:

sudo crictl logs <ContainerID>

Run shell on the running container:

sudo crictl exec -it <ContainerID> /bin/sh   

Stop Pod:

sudo crictl stopp <PodID>  

Remove Pod:

sudo crictl rmp <PodID> 

Remove Image:

sudo crictl rmi <ImageID>

Pulls image from dockerhub:docker.io:

sudo crictl pull alpine

Inspection of pods:

sudo crictl inspectp <PodID, e.g:4e2>