Docker cleanup for GitLab Runner

Question

I want to automatically cleanup all unused docker resources in order to release more disk space for next ci jobs. Therefore, I created a cron job to cleanup unused docker resources every 5 mins. My docker cleanup script is

sudo docker system prune --all --volumes --force

However, this cron job may mis-delete images which will be used later. For example, delete intermediate container images while docker build and lead to error unknown parent image ID.

Periodic cleanup doesn't seem to be a good practice. The better practice to me is execute docker cleanup script before/after gitlab runner executes a ci job.

To achieve this, I can set cleanup script in pre_build_script or post_build_script in gitlab runner. But, ci environment (container image) that executes build scripts is vary with ci job. The cleanup script may be incompatible with every os. pre_build_script and post_build_script are not suitable in my case.

Is there any approach to detect when gitlab runner starts to run a ci job? Or any better solutions to cleanup unused docker resources for every ci job?

Answer 1

Run docker-cleanup container as Runner CI service

I run a docker-cleanup container as CI service for each CI Job. My CI service is created in the beginning of CI Job and clean up unused docker resources. Problem solved!

Config.toml

...
[[runners]]
  ...
  [runners.docker]
  ...
    [[runners.docker.services]]
      name = "registry.gitlab.com/example/docker-cleanup:latest"
...

Docker-cleanup Dockerfile

FROM docker:stable

CMD ["docker", "system", "prune", "--volumes", "--all", "--force"]

It's better to use filter in docker system prune to avoid mis-deleting images