Prometheus - getsockopt: connection refused

Question

In the targets page of Prometheus, I'm getting the following error:

I'm using it in Linux host

Prometheus Version:

prometheus, version 1.1.2 (branch: master, revision: 36fbdcc30fd13ad796381dc934742c559feeb1b5)
  build user:       root@a74d279a0d22
  build date:       20160908-13:12:43
  go version:       go1.6.3

What is the issue here?

Answer 1

As the scrape error message says, Prometheus can't open a connection to your target at 172.19.36.189:9104. Can you connect to that IP/port manually? Are you perhaps running Prometheus in a container where it cannot reach that IP?

Answer 2

The connection may be failing anywhere in the traffic flow. This can help identify:

That said, if your service is running in the same host as Prometheus, you should try and use 127.0.0.1:9104/metrics as the endpoint in Prometheus scrape config, ensuring the service is indeed listening in port 9104.

As you try to troubleshoot and fix issues in each layer, check connectivity from your Prometheus host using a simple:

nc -v <service-ip> <service-port>