Docker Swarm: bypass load balancer and make direct request to specific containers

Question

I have two containers running in a swarm. Each exposes a /stats endpoint which I am trying to scrape.

However, using the swarm port obviously results in the queries being load balanced and therefore the stats are all intermingled:

+--------------------------------------------------+
|                       Server                     |
|    +-------------+             +-------------+   |
|    |             |             |             |   |
|    | Container A |             | Container B |   |
|    |             |             |             |   |
|    +-------------+             +-------------+   |
|                 \              /                 |
|                  \            /                  |
|                 +--------------+                 |
|                 |              |                 |
|                 | Swarm Router |                 |
|                 |              |                 |
|                 +--------------+                 |
|                         v                        |
+-------------------------|------------------------+
                          |                         
                       A Stats                      
                       B Stats                      
                       A Stats                      
                       B Stats                      
                          |                         
                          v                          

I want to keep the load balancer for application requests, but also need a direct way to make requests to each container to scrape the stats.

+--------------------------------------------------+
|                       Server                     |
|    +-------------+             +-------------+   |
|    |             |             |             |   |
|    | Container A |             | Container B |   |
|    |             |             |             |   |
|    +-------------+             +-------------+   |
|        |        \              /         |       |
|        |         \            /          |       |
|        |        +--------------+         |       |
|        |        |              |         |       |
|        |        | Swarm Router |         |       |
|        v        |              |         v       |
|        |        +--------------+         |       |
|        |                |                |       |
+--------|----------------|----------------|-------+
         |                |                |
      A Stats             |             B Stats
      A Stats       Normal Traffic      B Stats
      A Stats             |             B Stats
         |                |                |
         |                |                |
         v                |                v

A dynamic solution would be ideal, but since I don't intend to do any dynamic scaling something like hardcoded ports for each container would be fine:

::8080  Both containers via load balancer
::8081  Direct access to container A
::8082  Direct access to container B

Can this be done with swarm?

Answer 1

From inside an overlay network you can get IP-addresses of all replicas with tasks.<service_name> DNS query:

; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> -tA tasks.foo_test
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19860
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;tasks.foo_test.            IN  A

;; ANSWER SECTION:
tasks.foo_test.     600 IN  A   10.0.1.3
tasks.foo_test.     600 IN  A   10.0.1.5
tasks.foo_test.     600 IN  A   10.0.1.6

This is mentioned in the documentation.

Also, if you use Prometheus to scrape those endpoints for metrics, you can combine the above with dns_sd_configs to set the targets to scrape (here is an article how). This is easy to get running but somewhat limited in features (especially in large environments).

A more advanced way to achieve the same is to use dockerswarm_sd_config (docs, example configuration). This way the list of endpoints will be gathered by querying Docker daemon, along with some useful labels (i.e. node name, service name, custom labels).