Why does docker crash on high memory usage?

Question

I have a docker container that runs my REST service written in python Flask. I am running the container using VirtualBox on OSx.

This is the memory statistics on OSx at the time the container is started:

So, I have ~3gb of memory free. So I run my container with memory limit of 2 gb

docker run -d -m 2g --name mycontainer -p 5000:5000 foobar

Now I send ~100 REST requests to the service running on the container while at the same time running docker stats.

Eventually, the docker container crashes.

I am pasting below the data from docker stats right before the container crashes.

Crash 1: When running 100 different requests (Container crashes almost instantaneously.

CONTAINER           CPU %               MEM USAGE / LIMIT     MEM %               NET I/O               BLOCK I/O
27ee4ed4f98a        99.27%              256.9 MB / 2.147 GB   11.96%              163.2 kB / 7.958 kB   107.4 MB / 0 B
CONTAINER           CPU %               MEM USAGE / LIMIT   MEM %               NET I/O               BLOCK I/O
27ee4ed4f98a        99.77%              324 MB / 2.147 GB   15.09%              163.2 kB / 7.958 kB   107.4 MB / 0 B
CONTAINER           CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O
CONTAINER           CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O
CONTAINER           CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O

Crash 2: When running 1 request 100 times (Container crashes after around 30)

CONTAINER           CPU %               MEM USAGE / LIMIT     MEM %               NET I/O               BLOCK I/O
41fc484677fb        79.00%              891.5 MB / 2.147 GB   41.52%              12.13 MB / 429.8 kB   2.379 GB / 61.85 MB
CONTAINER           CPU %               MEM USAGE / LIMIT   MEM %               NET I/O               BLOCK I/O
41fc484677fb        85.83%              892 MB / 2.147 GB   41.54%              12.13 MB / 429.8 kB   3.071 GB / 61.85 MB
CONTAINER           CPU %               MEM USAGE / LIMIT   MEM %               NET I/O               BLOCK I/O
41fc484677fb        85.83%              892 MB / 2.147 GB   41.54%              12.13 MB / 429.8 kB   3.071 GB / 61.85 MB
CONTAINER           CPU %               MEM USAGE / LIMIT   MEM %               NET I/O               BLOCK I/O
41fc484677fb        86.01%              892 MB / 2.147 GB   41.54%              12.13 MB / 429.8 kB   3.81 GB / 61.85 MB
CONTAINER           CPU %               MEM USAGE / LIMIT   MEM %               NET I/O               BLOCK I/O
41fc484677fb        86.01%              892 MB / 2.147 GB   41.54%              12.13 MB / 429.8 kB   3.81 GB / 61.85 MB
CONTAINER           CPU %               MEM USAGE / LIMIT     MEM %               NET I/O               BLOCK I/O
41fc484677fb        86.28%              892.2 MB / 2.147 GB   41.55%              12.13 MB / 429.8 kB   4.508 GB / 61.85 MB
CONTAINER           CPU %               MEM USAGE / LIMIT     MEM %               NET I/O               BLOCK I/O
41fc484677fb        86.28%              892.2 MB / 2.147 GB   41.55%              12.13 MB / 429.8 kB   4.508 GB / 61.85 MB
CONTAINER           CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O
CONTAINER           CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O
CONTAINER           CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O

docker ps -a shows the following after the crash

CONTAINER ID        IMAGE                      COMMAND             CREATED             STATUS                       PORTS               NAMES
41fc484677fb        foobar   "python service.py"    7 minutes ago       Exited (137) 2 minutes ago                       mycontainer

running dmesg shows several out of memory errors:

➜  ~ docker exec -it mycontainer dmesg | grep "Out of memory"
Out of memory: Kill process 2006 (python) score 872 or sacrifice child
Out of memory: Kill process 2496 (python) score 873 or sacrifice child
Out of memory: Kill process 2807 (python) score 879 or sacrifice child
Out of memory: Kill process 3101 (python) score 875 or sacrifice child
Out of memory: Kill process 5393 (python) score 868 or sacrifice child
Out of memory: Kill process 5647 (python) score 868 or sacrifice child
Out of memory: Kill process 5926 (python) score 877 or sacrifice child
Out of memory: Kill process 6328 (python) score 873 or sacrifice child
Out of memory: Kill process 7923 (python) score 872 or sacrifice child
Out of memory: Kill process 10183 (python) score 873 or sacrifice child

Question

1. How can I avoid crashes like these?

2. This is just on my local machine but eventually I'm planning to deploy this container to production. What approaches should I follow to protect against crashes? Should I stand multiple clones of this container behind a Nginx load balancer?

3. In production, I'm planning to run single container on a single server. If I run single container on a server and don't run anything else on that server, will the container be able to use all the computing resources available to it?

Answer 1

Welcome to the wonderful world of resources :)

Putting a limit on a container doesn't make you stay under the limit, it just tells the kernel when to start squeezing you and when to kill you. You have to actually stay below your limit. In many cases this means watching your memory footprint and queuing or dropping requests when you can't satisfy them within your budget. AKA load-shedding.

The upside, though, is that you now have a pretty clear signal when you need more replicas of your containers.