Prometheus with several servers

Question

I am starting to look at the Prometheus project and prototype it. So far I have implemented a few counters in my application (running on Weblogic), exposed the metrics servlet (dedicated war file), started on a dedicated machine a Prometheus server + Grafana using Docker images. Everything's working fine, the server scraps the counters and Grafana displays them in graphs.

But I am facing an issue because of my runtime environment. Basically I target my application via https://<myserver>/<myproduct>, and target the metrics via https://<myserver>/prometheus/metrics/, but then behind the scene I can hit two different Weblogic servers, that are not exposed outside of the platform they are deployed in (so my machine hosting the Prometheus server cannot reach the Weblogic instances directly). So I get different metrics depending on the server that will chosen to be hit by the load balancer.

Is there a way to workaround this kind of setup? A way to tell Prometheus server to aggregate the results somehow?

Thanks for your help!

Answer 1

Prometheus needs to talk to each instance directly, so going via any form of load balancer won't work. You should see if there's a way to have direct access.