Why does creating a Docker network randomly breaks all connections from my Jenkins server?

Question

I am using Jenkins to spin a docker-compose project on another server and run tests. The server where Docker is installed is not on the same sub-network as the Jenkins server. We use a different server because the latter is provided by a third party without the ability to spin containers.

The Docker server is a Red Hat 7.7 and it's running Docker CE 19.03.1.

Jenkins connects to the Docker server in ssh, then runs a docker-compose command. Docker creates three containers and a default network for the project.

Everything works as expected, but once in while, Jenkins loses all connectivity to the Docker server. After spinning up the containers, the current job stalls until it times out, usually with a "broken pipe" error.
From then on, Jenkins cannot connect to the Docker server at all. All connections on any port simply time out.
Until I connect to it from my PC, which I for some reason still can, and manually stop and remove the containers and the network. Then everything works again.

The issue really seems to be that Docker network. But why would it only break connections from the Jenkins server and not my PC? And why randomly?

I am aware this may be too vague or too specific for SO. I am not even sure where to start debugging. Please let me know in the comments what information I should add, or if I should ask another community.

Answer 1

docker is subject to create networks that conflict with your existing networks. For example:

Traffic from 10.0.0.0/16 to 10.1.0.0/16 and back is routed over the peering connection (default gateway).

If docker running on the instance in the 10.0.0.0/16 subnet creates a bridged network (i.e. br0) with the CIDR 10.1.0.0 then you'll also get a local route for that traffic.

Trying to connect from 10.0.0.0/16 to 10.1.0.0/16 will be routed to the docker network and not the default gateway preventing you from connecting to your instance(s) in the 10.1.0.0/16 subnet.

You can create the docker network with a fixed subnet / gateway to avoid these issues i.e.: docker network create --subnet=172.128.28.0/24 br0.