failed to garbage collect required amount of images. Wanted to free 6283487641 bytes, but freed 0 bytes

Question

I have searched many websites and articles but not found any perfect answer. I am using eks version 1.18. I can see a few of the pods are "Evicted", but when trying to check the node I can see the error "(combined from similar events): failed to garbage collect required amount of images. Wanted to free 6283487641 bytes, but freed 0 bytes".

Is there any way we can find the reason why it's failing? or how to fix this issue? Any suggestions are most welcome.

I can see the disk "overlays" filesystem is almost full within a few hours. I am not sure what's going on. The below screenshot shows my memory utilization.

Answer 1

So a workaround that can stabilize situation for a while (giving your time to mount a larger volume for storing images) is to start using local images cache, by changing imagePullPolicy in your Deployment (or Pod) manifest to:

spec.containers.imagePullPolicy: "ifNotPresent"

---

One situation I've encountered where such quick storage exhaustion can happen is when you set imagePullPolicy to Always and then the image fails to pull completely (one reason being that there is not enough space). k8s then enters into an image pull loop (not throttled sufficiently by the backoff mechanism) and those unique incomplete image parts with different checksums combined with the "always pull" request can quickly consume all available storage dedicated to docker images (on the partition where containerd is located).