Docker creating too many images

Question

I have several projects that are in their own docker projects. I've noticed over time that when I need to setup a new environment, it takes longer and longer to complete. It seems like each version is anywhere from 25-100mb and it downloads all of them every time.

What could be some things causing this, and is there a better way to do it?

Pulling repository private.repo.com:8080/project
24e73aa61b9e: Pulling dependent layers
e9e06b06e14c: Download complete
a82efea989f9: Download complete
37bea4ee0c81: Download complete
07f8e8c5e660: Download complete
eebb8b8af862: Download complete
66dfa7610c51: Download complete
be9344bb178f: Download complete
f9b0b5922fcc: Download complete
d538871b66d9: Download complete
63b83602dd59: Download complete
4701d9c2b782: Download complete
e0867454dc78: Download complete
5f8f0c295506: Download complete
ff20a42d5239: Download complete
79093a41f7a4: Download complete
f67f10a60f8f: Download complete
9ae271a15a20: Download complete
b8e39263e460: Download complete
cd92a7abfb56: Download complete
3ab53402c585: Download complete
e6fbd8fbe1c8: Download complete
6b741d246738: Download complete
5d2de83d20bf: Download complete
72c062ac5a98: Download complete
8b9f7ca662b7: Download complete
10e830b030bf: Download complete
2f9c03ddb7f7: Download complete
3a7a81509db4: Download complete
11f4dab16ebd: Downloading [=======>                                           ] 15.68 MB/106 MB

Answer 1

You have too many commands in your Dockerfiles. Each command — RUN, ENV, EXPOSE, etc. — creates a new layer in the resulting image, and each of those layers must be downloaded in order to download the complete image. Try reducing the number of layers by combining multiple RUN commands into one RUN with &&.

Answer 2

Don't have reputations to add comments, hence adding as a answer. In addition to @jwodder's answer, you should also keep in mind that when you are writing the Dockerfiles for generating images, the order of commands also plays important role.

Assume you have a 15 commands in your dockerfile for generating one project's docker image.

To generate another project's image you added just one different command between the 3rd and 4th commands --> this would result in total of 28 layers in the file system (15 layers for the first project + (16 -3 [cache/ reused the first three layers from the first project]).

This is because, Docker always creates a layer for each command and checks the cache if there is an immage with the current command + previous cache layer hash match. If any of them doesn't match, it would endup creating a new image. Hence if there is just one command introduced in between, all the image layers subsequent to that will get invalidated and Docker creates new layers for those for the subsequent build.

This is also applicable to the same project and changing the version attribute may be in the first few commands. This would result in invalidating all the previously created cache layers and for the same project now there will be more layers created.