Docker starts unknown container (buildx_buildkit_default) and it is crashed

Question

I have 4-5 services running by docker in my Ubuntu server. Everything has worked perfectly for a couple months until last night. I was trying to re-run my service and this happened.

[+] Building 3.3s (0/1)                                                                                                   
 => [internal] booting buildkit                                                                                      3.3s
 => => starting container buildx_buildkit_default

and it shows this error,

 => ERROR [internal] booting buildkit                                                                               14.8s
 => => starting container buildx_buildkit_default                                                                   14.8s
------                                                                                                                    
 > [internal] booting buildkit:
#0 14.76 time="2023-10-31T02:28:51Z" level=warning msg="using host network as the defaultime="2023-10-31T02:28:51Z" level=warning msg="using host network as the default"
#0 14.76 time="2023-10-31T02:28:51Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
#0 14.76 dtime="2023-10-31T02:28:51Z" level=info msg="found 1 workers, default=\"v0xww56tjtkxcr6ytuye297m3\""
#0 14.76 `time="2023-10-31T02:28:51Z" level=warning msg="currently, only the default worker can be used."
#0 14.76 \time="2023-10-31T02:28:51Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
#0 14.76 =time="2023-10-31T02:29:33time="2023-10-31T02:28:51Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
#0 14.76 time="2023-10-31T02:28:51Z" level=warning msg="currently, only the default worker can be used."
#0 14.76 time="2023-10-31T02:28:51Z" level=warning msg="currently, only the default worker can be used."
#0 14.76 time="2023-10-31T02:29:33Z" level=info msg="stopping server"
#0 14.76 buildkitd: context canctime="2023-10-31T02:29:33Z" level=info msg="stopping server"
#0 14.76 time="2023-10-31T02:29:40Z" time="2023-10-31T02:29:40Z" level=info msg="auto snapshotter: using overlayfs"
#0 14.76 time="2023-10-31T02:29:40Z" level=info msg="found worker \"v0xww56tjtkxcr6ytuye297time="2023-10-31T02:29:40Z" level=info msg="found worker \"v0xww56tjtkxcr6ytuye297m3\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:51e49fc22124 org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.oci.process-mode:sandbox org.mobyproject.buildkit.worker.selinux.enabled:false org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/amd64/v2 linux/amd64/v3 linux/386]"
#0 14.76 time="2023-10-31T02:29:40Z" level=info msg="found 1 workers, default=\"v0xww56tjtkxcr6ytuye297m3\""
#0 14.76 `time="2023-10-31T02:29time="2023-10-31T02:29:40Z" level=info msg="found 1 workers, default=\"v0xww56tjtkxcr6ytuye297m3\""
#0 14.76 time="2023-10-31T02:29:40Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
#0 14.76 time="2023-10-31T02:29:40Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
#0 14.76 buildkitd: context canceled
#0 14.76 Otime="2023-10-31T03:11:41buildkitd: context canceled
#0 14.76 time="2023-10-31T03:11:41Z" level=warning msg="using host network as the defaultime="2023-10-31T03:11:41Z" level=warning msg="using host network as the default"
#0 14.76 time="2023-10-31T03:11:41Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
#0 14.76 dtime="2023-10-31T03:11:41Z" level=info msg="found 1 workers, default=\"v0xww56tjtkxcr6ytuye297m3\""
#0 14.76 `time="2023-10-31T03:11:41Z" level=warning msg="currently, only the default worker can be used."
#0 14.76 \time="2023-10-31T03:11:41Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
#0 14.76 =time="2023-10-31T03:12:30time="2023-10-31T03:11:41Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
#0 14.76 time="2023-10-31T03:11:41Z" level=warning msg="currently, only the default worker can be used."
#0 14.76 time="2023-10-31T03:11:41Z" level=warning msg="currently, only the default worker can be used."
#0 14.76 time="2023-10-31T03:12:30Z" level=info msg="stopping server"
#0 14.76 buildkitd: context canctime="2023-10-31T03:12:30Z" level=info msg="stopping server"
#0 14.76 time="2023-10-31T03:12:46Z" time="2023-10-31T03:12:46Z" level=info msg="auto snapshotter: using overlayfs"
#0 14.76 time="2023-10-31T03:12:46Z" level=info msg="found worker \"v0xww56tjtkxcr6ytuye297time="2023-10-31T03:12:46Z" level=info msg="found worker \"v0xww56tjtkxcr6ytuye297m3\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:51e49fc22124 org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.oci.process-mode:sandbox org.mobyproject.buildkit.worker.selinux.enabled:false org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/amd64/v2 linux/amd64/v3 linux/386]"
#0 14.76 time="2023-10-31T03:12:46Z" level=info msg="found 1 workers, default=\"v0xww56tjtkxcr6ytuye297m3\""
#0 14.76 `time="2023-10-31T03:12time="2023-10-31T03:12:46Z" level=info msg="found 1 workers, default=\"v0xww56tjtkxcr6ytuye297m3\""
#0 14.76 time="2023-10-31T03:12:46Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
#0 14.76 time="2023-10-31T03:12:46Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
#0 14.76 buildkitd: context canceled
#0 14.76 Otime="2023-10-31T03:18:56buildkitd: context canceled
#0 14.76 time="2023-10-31T03:18:56Z" level=warning msg="using host network as the defaultime="2023-10-31T03:18:56Z" level=warning msg="using host network as the default"
#0 14.76 time="2023-10-31T03:18:56Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
#0 14.76 dtime="2023-10-31T03:18:56Z" level=info msg="found 1 workers, default=\"v0xww56tjtkxcr6ytuye297m3\""
#0 14.76 `time="2023-10-31T03:18:56Z" level=warning msg="currently, only the default worker can be used."
#0 14.76 \time="2023-10-31T03:18:56Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
#0 14.76 time="2023-10-31T03:18:56Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
#0 14.76 time="2023-10-31T03:18:56Z" level=warning msg="currently, only the default worker can be used."
#0 14.76 time="2023-10-31T03:18:56Z" level=warning msg="currently, only the default worker can be used."
#0 14.76 

Then I try to run my docker service again. It just show this,

[+] Building 0.0s (0/0)                                                                                                   
http: invalid Host header

I check docker ps and there is a buildx_buildkit_default running. If I stop it, it will cause the above error again.

Ps. There is no any changes on the Dockerfile. This has happened from nowhere.

Answer 1

Check out this thread: https://github.com/docker/compose/issues/11154

If you have installed Docker via Snap then upgrade your docker compose with the below command:

sudo snap refresh docker --channel=latest/edge

It fixed this issue for me.

More context:

On Oct 30, docker upgraded to version 20.10.24 which enabled buildx; however, if docker compose or docker-compose is still old then the above error happens.

Previous:

$ docker compose version
Docker Compose version v2.17.2

After upgrade:

$ docker compose version
Docker Compose version v2.20.3

Everything worked fine after upgrading with sudo snap refresh docker --channel=latest/edge