Docker pull an image after removing it, all layers except the last one already exist

Question

Before pulling, the information in docker images:

REPOSITORY                    TAG                 IMAGE ID            CREATED             SIZE
weaveworks/weave-kube         2.5.2               f04a043bb67a        7 months ago        148MB
weaveworks/weave-npc          2.5.2               5ce48e0d813c        7 months ago        49.6MB
k8s.gcr.io/kube-proxy-amd64   v1.11.0             1d3d7afd77d1        18 months ago       97.8MB
k8s.gcr.io/pause              3.1                 da86e6ba6ca1        2 years ago         742kB

However, when I tried to download an image, it shows that all layers except the last one already exist:

$ docker pull image_name:s2-audio-1
s2-audio-1: Pulling from image_name:s2-audio-1
121er07f4eba: Already exists
0c8f4rew94dc: Already exists
198ef12987c0: Already exists
1pp1b5eaa295: Already exists
8a4fm1842a93: Already exists
3446a509a6e7: Already exists
10h6cedf3c5e: Already exists
0te381950737: Already exists
6gdf14004bd1: Already exists
88le96f02ad0: Already exists
udhb049fbae0: Already exists
llae95975878: Already exists
9fl1d0e8bd4e: Already exists
jd3b2c426fa1: Already exists
6j10b5462122: Already exists
21c258fef3cc: Already exists
0jzb3ee699a9: Already exists
zj019b962h6a: Already exists
04345388691c: Already exists
eyu517086e97: Pull complete
Digest: sha256:9wk43b75202a889711817b52e5979fg33ed387727a7a6eddc8e521d1da05ld2a
Status: Downloaded newer image for image_name:s2-audio-1

I guess there are some problems with docker state, so I tried to solve the problem by:

sudo service docker stop
sudo rm -rf /var/lib/docker
sudo service docker start

Then I can pull image_name:s2-audio-1 normally.

s2-audio-1: Pulling from image_name:s2-audio-1
121er07f4eba: Pull complete
0c8f4rew94dc: Pull complete
198ef12987c0: Pull complete
1pp1b5eaa295: Pull complete
8a4fm1842a93: Pull complete
3446a509a6e7: Pull complete
10h6cedf3c5e: Pull complete
0te381950737: Pull complete
6gdf14004bd1: Pull complete
88le96f02ad0: Pull complete
udhb049fbae0: Pull complete
llae95975878: Pull complete
9fl1d0e8bd4e: Pull complete
jd3b2c426fa1: Pull complete
6j10b5462122: Pull complete
21c258fef3cc: Pull complete
0jzb3ee699a9: Pull complete
zj019b962h6a: Pull complete
04345388691c: Pull complete
eyu517086e97: Pull complete
Digest: sha256:9wk43b75202a889711817b52e5979fg33ed387727a7a6eddc8e521d1da05ld2a
Status: Downloaded newer image for image_name:s2-audio-1

However, when I tried to remove this image:

Untagged: image_name:s2-audio-1
Untagged: image_name@sha256:9wk43b75202a889711817b52e5979fg33ed387727a7a6eddc8e521d1da05ld2a
Deleted: sha256:d7349f1dd8yhdffb55b9c68ec2f4573e6e45d6f22bd065e940d6e8f06627f586
Deleted: sha256:2da3218b4cd1a85e388k4b435411ac1082ce23571eeea4570a230583ee34795a

The number of untagged and deleted layers is much less than before. It shuld be like this:

$ docker rmi image_name:s2-audio-3
Untagged: image_name:s2-audio-3
Deleted: sha256:eds287df3d80dc048096dd67ecf09008b99dbe23ht8a6d4244e82ht9e0620c6d
Deleted: sha256:0d387c6877f0382a8d62474f1fdc3alo44cb1159fdcd3b7fdd9fb0c4f61d32a8
Deleted: sha256:d1b2f94b32ec9a4be182fcb4ef4919f439v6b6765975295654a3e594daa24646
Deleted: sha256:2f9uug161f8c01723367a21121797400261d41426a0dbd2a1c85828df7851bb8
Deleted: sha256:a6865f228367ce14548da56a9d9bbe5568sf6b30eb6de04bf46a758356d93280
Deleted: sha256:fc47aad0131dca88bd2ads792f238c577594a7b0e27f678cea0e64f447c94b44
Deleted: sha256:11e5f8ad0b20ebaa39c17273a859wr32a9260dafea5e513850ec56106ede679a
Deleted: sha256:cabf7d90af0bew5c5ce4965a8e994146fed9114f32b3da9bab321963c63ea308
Deleted: sha256:c7cfdc043db5f78arfg7a660f60642d8d6f4de770c10089f3dc5b3d989e856dd
Deleted: sha256:30a82effba73451b0c235859724995cebfe3g6574d286ffe5c2beb74150b977f
Deleted: sha256:7eae8b82dd96e5bde4c23a5868222c7eae05f2743c608d44dffdfbed54c56ef0
Deleted: sha256:b9ca36ae1e9e7e201802597f6e434adc5533sdsc90d9ab0476502b1cf5bd5ee4
Deleted: sha256:a765c11dsbbfc69f1089f96e55ed02a901e35c1f2834cebc790a2046fe5dcaf1

Then I tried to pull image_name:s2-audio-1 again, the pulling problem reoccurred. (i.e. all layers except the last one already exist.)

Now, Docker only removes and untags one layer every time I remove whatever image. The remaining layers are still saved in the disk and invisible even though I use docker images:

REPOSITORY                    TAG                 IMAGE ID            CREATED             SIZE
weaveworks/weave-kube         2.5.2               f04a043bb67a        7 months ago        148MB
weaveworks/weave-npc          2.5.2               5ce48e0d813c        7 months ago        49.6MB
k8s.gcr.io/kube-proxy-amd64   v1.11.0             1d3d7afd77d1        18 months ago       97.8MB
k8s.gcr.io/pause              3.1                 da86e6ba6ca1        2 years ago         742kB

This worries me that the invisible layers consume a lot of my storage, and there is no proper way to delete them. (sudo rm -rf /var/lib/docker all the time is not a good idea.)

Answer 1

Running docker image rm <image> (or the equivalent docker rmi <image>) really just untags the image and removes the final layer. Other layers will be left in place because they could potentially be re-used by other images.

If you want to remove these "dangling" layers (layers that are no longer used by a tagged image), you can use the docker image prune command.

You can list dangling layers by running:

docker image ls --filter dangling=true