docker DNS configuration is not updated after connecting to VPN

Question

Question

I'm using docker 19.03.9, on Ubuntu 18.04; docker daemon is configured to start automatically. When I connect to my employer VPN, docker DNS configuration is not updated, unless I restart the docker daemon.

Is there a way to update automatically docker DNS configuration when host DNS configuration is changing?

Investigation

My scenario: I want to ping a server inside my company network (let's say server.mycompany.com) from home. I'm trying to ping it from my host (laptop running Ubuntu), and then from inside a container on my host

Note: all IP addresses are masked in examples below

I've created a small docker image to test my scenario:

FROM ubuntu:18.04

RUN apt-get update \
    && apt-get install -yqq iputils-ping

CMD cat /etc/resolv.conf \
    && echo \
    && ping -c 2 server.mycompany.com

Then I've built this image: docker build -t test_dns .

1- test without VPN connection

In a terminal on my Ubuntu laptop, I execute the 3 commands below:

    $ cat /etc/resolv.conf
    # Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
    #     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
    # 127.0.0.53 is the systemd-resolved stub resolver.
    # run "systemd-resolve --status" to see details about the actual nameservers.

    nameserver 127.0.0.53
    search home

    $ ping server.mycompany.com
    ping: server.mycompany.com: Name or service not known

    $ docker run test_dns
    # This file is managed by man:systemd-resolved(8). Do not edit.
    #
    # This is a dynamic resolv.conf file for connecting local clients directly to
    # all known uplink DNS servers. This file lists all configured search domains.
    #
    # Third party programs must not access this file directly, but only through the
    # symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
    # replace this symlink by a static file or a different symlink.
    #
    # See man:systemd-resolved.service(8) for details about the supported modes of
    # operation for /etc/resolv.conf.

    nameserver 192.168.xxx.xxx
    nameserver 198.235.yyy.yyy
    search home

    ping: server.mycompany.com: Name or service not known

As expected, server.mycompany.com can not be resolved and is not reachable (neither from my host nor from a container)

Note that:

• 192.168.xxx.xxx is my wifi router IP address
• 198.235.yyy.yyy is my ISP DNS server address

2- After connecting to my company VPN

Once connected to mycompany VPN, I execute the 3 same commands:

    $ cat /etc/resolv.conf
    # Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
    #     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
    # 127.0.0.53 is the systemd-resolved stub resolver.
    nameserver 10.xxx.xxx.xxx
    nameserver 10.yyy.yyy.yyy
    nameserver 10.zzz.zzz.zzz
    nameserver 127.0.0.53

    search mycompany.com mycompany-other-domain.com home

    $ ping -c 2 server.mycompany.com
    PING server.mycompany.com (10.sss.sss.sss) 56(84) bytes of data.
    64 bytes from 10.sss.sss.sss (10.sss.sss.sss): icmp_seq=1 ttl=120 time=15.6 ms
    64 bytes from 10.sss.sss.sss (10.sss.sss.sss): icmp_seq=2 ttl=120 time=16.4 ms

    --- server.mycompany.com ping statistics ---
    2 packets transmitted, 2 received, 0% packet loss, time 1000ms
    rtt min/avg/max/mdev = 15.681/16.058/16.435/0.377 ms

    $ docker run test_dns
    # This file is managed by man:systemd-resolved(8). Do not edit.
    #
    # This is a dynamic resolv.conf file for connecting local clients directly to
    # all known uplink DNS servers. This file lists all configured search domains.
    #
    # Third party programs must not access this file directly, but only through the
    # symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
    # replace this symlink by a static file or a different symlink.
    #
    # See man:systemd-resolved.service(8) for details about the supported modes of
    # operation for /etc/resolv.conf.

    nameserver 192.168.xxx.xxx
    nameserver 198.235.yyy.yyy
    search home

    ping: server.mycompany.com: Name or service not known

• on my host:

  Connecting to mycompany VPN has updated DNS configuration: we can see mycompany 3 private DNS servers (10.xxx.xxx.xxx, 10.yyy.yyy.yyy and 10.zzz.zzz.zzz)

  As expected, server.mycompany.com is resolved and reached by my ping request

• in a container:

  Docker is still using previous DNS configuration. It was not updated with host DNS changes (when connecting to VPN).

  Is there a way to make docker update its DNS configuration automatically (when it changes on host)?

3- Restarting docker daemon (still connected to my company VPN)

I restart docker daemon: sudo service docker restart

Then I execute the 3 same commands again:

    $ cat /etc/resolv.conf
    # Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
    #     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
    # 127.0.0.53 is the systemd-resolved stub resolver.
    nameserver 10.xxx.xxx.xxx
    nameserver 10.yyy.yyy.yyy
    nameserver 10.zzz.zzz.zzz
    nameserver 127.0.0.53

    search mycompany.com mycompany-other-domain.com home

    $ ping -c 2 server.mycompany.com
    PING server.mycompany.com (10.sss.sss.sss) 56(84) bytes of data.
    64 bytes from 10.sss.sss.sss (10.sss.sss.sss): icmp_seq=1 ttl=120 time=23.3 ms
    64 bytes from 10.sss.sss.sss (10.sss.sss.sss): icmp_seq=2 ttl=120 time=11.7 ms

    --- server.mycompany.com ping statistics ---
    2 packets transmitted, 2 received, 0% packet loss, time 999ms
    rtt min/avg/max/mdev = 11.786/17.551/23.317/5.767 ms

    $ docker run test_dns
    # Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
    #     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
    # 127.0.0.53 is the systemd-resolved stub resolver.
    nameserver 10.xxx.xxx.xxx
    nameserver 10.yyy.yyy.yyy
    nameserver 10.zzz.zzz.zzz
    search mycompany.com mycompany-other-domain.com home

    PING server.mycompany.com (10.sss.sss.sss) 56(84) bytes of data.
    64 bytes from 10.sss.sss.sss (10.sss.sss.sss): icmp_seq=1 ttl=119 time=11.5 ms
    64 bytes from 10.sss.sss.sss (10.sss.sss.sss): icmp_seq=2 ttl=119 time=10.7 ms

    --- server.mycompany.com ping statistics ---
    2 packets transmitted, 2 received, 0% packet loss, time 1001ms
    rtt min/avg/max/mdev = 10.750/11.159/11.569/0.422 ms

• on my host: still working perfectly: server.mycompany.com is still resolved and reachable

• in a container: docker DNS configuration now uses mycompany VPN settings; server.mycompany.com is now resolved and reached by my ping request inside any docker container.

4- Disconnect from my company VPN

I disconnect from mycompany VPN, and execute the 3 same commands again:

    $ cat /etc/resolv.conf
    # Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
    #     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
    # 127.0.0.53 is the systemd-resolved stub resolver.
    nameserver 127.0.0.53

    search home

    $ ping -c 2 server.mycompany.com
    ping: server.mycompany.com: Name or service not known

    $ docker run test_dns
    # Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
    #     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
    # 127.0.0.53 is the systemd-resolved stub resolver.
    search home

    nameserver 8.8.8.8
    nameserver 8.8.4.4

    ping: server.mycompany.com: Name or service not known

• on my host: exact same behavior as it was at step 1: I'm outside mycompany network, an thus server.mycompany.com can neither be resolved nor reached

• in a container: this time, DNS configuration has been automatically updated to use Google public DNS servers. server.mycompany.com can neither be resolved nor reached (as expected)

5- Connect again to my company VPN

I connect again to mycompany VPN, and execute the 3 same commands again:

    $ cat /etc/resolv.conf
    # Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
    #     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
    # 127.0.0.53 is the systemd-resolved stub resolver.
    nameserver 10.xxx.xxx.xxx
    nameserver 10.yyy.yyy.yyy
    nameserver 10.zzz.zzz.zzz
    nameserver 127.0.0.53

    search mycompany.com mycompany-other-domain.com home

    $ ping -c 2 server.mycompany.com
    PING server.mycompany.com (10.sss.sss.sss) 56(84) bytes of data.
    64 bytes from 10.sss.sss.sss (10.sss.sss.sss): icmp_seq=1 ttl=120 time=20.5 ms
    64 bytes from 10.sss.sss.sss (10.sss.sss.sss): icmp_seq=2 ttl=120 time=12.3 ms

    --- server.mycompany.com ping statistics ---
    2 packets transmitted, 2 received, 0% packet loss, time 1001ms
    rtt min/avg/max/mdev = 12.308/16.442/20.577/4.136 ms

    $ docker run test_dns
    # Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
    #     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
    # 127.0.0.53 is the systemd-resolved stub resolver.
    nameserver 10.xxx.xxx.xxx
    nameserver 10.yyy.yyy.yyy
    nameserver 10.zzz.zzz.zzz
    search mycompany.com mycompany-other-domain.com home

    PING server.mycompany.com (10.sss.sss.sss) 56(84) bytes of data.
    64 bytes from 10.sss.sss.sss (10.sss.sss.sss): icmp_seq=1 ttl=119 time=13.2 ms
    64 bytes from 10.sss.sss.sss (10.sss.sss.sss): icmp_seq=2 ttl=119 time=14.3 ms

    --- server.mycompany.com ping statistics ---
    2 packets transmitted, 2 received, 0% packet loss, time 1001ms
    rtt min/avg/max/mdev = 13.257/13.791/14.325/0.534 ms

• on my host: DNS configuration has been changed when connecting again to VPN; thus server.mycompany.com can be resolved and reached again

• in a container: this time, DNS configuration has been automatically updated to use mycompany DNS servers!! server.mycompany.com can be resolved and reached.

6- Conclusion and questions:

• Why docker is behaving differently in steps 2 and 5 ? What does make these steps different?
• Is there a way to make docker update its DNS configuration automatically in step 2?
• Is this a bug in docker daemon?

Answer 1

1. Stop docker service: sudo systemctl stop docker (or if you have docker installed via snap: sudo snap stop docker)
2. Restart systemd-resolved: sudo service systemd-resolved restart
3. Start docker service again: sudo systemctl start docker (or snap: sudo snap start docker).

I'm on Ubuntu 20.04 and tried everything (from adding docker dns arguments in DOCKER_OPTS to deleting the docker0 network interface and re-creating) but what ended up finally solving it though was the above steps.

After that hostnames that could only be resolved behind the VPN I connect to were being resolved inside docker containers running and during the docker build step.