Docker CannotCreateContainerError: Thin Pool has 0 free data blocks

Question

I am trying to run a Nextflow pipeline using AWS (an EC2 instance) which requires using docker, but the following error appears:

CannotCreateContainerError: Error response from daemon: devmapper: Thin Pool has 0 free data blocks which is less than minimum required 4449 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior

And after finding this error my pipeline completely dies. The most recurrent answer to this problem that I have found online is to do a docker system prune, so I can free some space, but after doing that the error persists, and free data blocks are still 0.

My guess is that I am not being able to acces to the data blocks, but as it is my first time working with Docker, I am completely lost.

In case it is interesting, if I run docker info:

Client:
 Debug Mode: false

Server:
 Containers: 4
  Running: 0
  Paused: 0
  Stopped: 4
 Images: 22
 Server Version: 19.03.13-ce
 Storage Driver: devicemapper
  Pool Name: docker-docker--pool
  Pool Blocksize: 524.3kB
  Base Device Size: 536.9GB
  Backing Filesystem: ext4
  Udev Sync Supported: true
  Data Space Used: 14.55GB
  Data Space Total: 23.33GB
DOCKER_STORAGE_OPTIONS="--storage-driver devicemapper --storage-opt dm.thinpooldev=/dev/mapper/docker-docker--pool --storage-opt dm.use_deferred_removal=true --storage-opt dm.use_deferred_deletion=true   Data Space Available: 8.782GB
  Metadata Space Used: 4.891MB
  Metadata Space Total: 25.17MB
  Metadata Space Available: 20.28MB
/*
  Thin Pool Minimum Free Space: 2.333GB
  Deferred Removal Enabled: true
  Deferred Deletion Enabled: true
  Deferred Deleted Device Count: 0
  Library Version: 1.02.135-RHEL7 (2016-11-16)
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: c623d1b36f09f8ef6536a057bd658b3aa8632828
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0 (expected: fec3683)
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.14.225-121.362.amzn1.x86_64
 Operating System: Amazon Linux AMI 2018.03
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 985.5MiB
 Name: ip-172-31-33-79
 ID: QBVF:B7D5:3KRH:3BYR:UU27:XEUW:RWLE:SLAW:F6AG:LKD2:FD3E:LHLQ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Any clue about how to solve this issue?

Answer 1

Looking at your docker info above, I noticed two things:

1. Your Docker storage driver (devicemapper) is deprecated, and
2. Your Amazon Linux AMI is also deprecated.

I think if you use the newer Amazon Linux 2 AMI, your new Docker would use the overlay2 storage driver, which is the preferred storage driver for Docker. You shouldn't need to upgrade, but it might be the easiest thing to try unless you're tied to this instance.