no space left on device, which is unexpected. MountVolume.SetUp failed for volume in kubernetes cluster

Question

Today my kubernetes cluster(v1.15.2) node disk full and cause pods give this tips:

Update plugin resources failed due to failed to write checkpoint file "kubelet_internal_checkpoint": write /var/lib/kubelet/device-plugins/.261578065: no space left on device, which is unexpected.
MountVolume.SetUp failed for volume "default-token-xnrwt" : mkdir /opt/k8s/k8s/kubelet/pods/67eaa71b-adf4-4365-a1c7-42045d5e9426: no space left on device

I login into server and find the disk usage is 100%, so I remove some log file and release 10GB + disk space, but now it seems pod is not recovery automaticlly and still have this error tips:

what should I do to fix this problem? I am try to restart all pods, and all pods works fine. But finally I found the error tips message still give me tips no space left and did not disappear automaticlly. I check the node status and find the node has no disk pressure. How to make the error tips disappear?

Answer 1

Other possibility is incorrect unit values for resource requests/limits (ex, using mi instead of Mi).

For example:

apiVersion: v1
kind: Pod
spec:
  containers:
    - name: {container_name}
      resources:
        limits:
          memory: "512mi" # incorrect; should be "512Mi"
          cpu: "200m"

Answer 2

Posting this Community Wiki as a solution was mentioned in the comment section.

Errors like no space left on device, which is unexpected. and no space left on device occurs when your application is using 100% of available space. You can check it using the command $ df -h.

Solution

To resolve this kind of issue, you have to "make some space" in volume. You can do it by manual removal of files (OP did it in this scenario).
Once you make some space you should restart kubelet using $ systemctl restart kubelet.

Above steps resolved the OPs issue.

In addition, in some specific scenarios you might also restart docker service using $ service docker restart or specific resource.