Prometheus cannot scrape from spring-boot application over HTTPS

Question

I'm deploying a spring-boot application and prometheus container through docker, and have exposed the spring-boot /actuator/prometheus endpoint successfully. However, when I enable prometheus debug logs, I can see it fails to scrape the metrics:

ts=2022-02-02T03:54:46.210Z
caller=scrape.go:1292
level=debug
component="scrape manager"
scrape_pool=spring-actuator
target=https://127.0.0.1:8443/actuator/prometheus/
msg="Scrape failed"
err="Get \"https://127.0.0.1:8443/actuator/prometheus/\": dial tcp 127.0.0.1:8443: connect: connection refused"

I'm thinking it's something to do with how I've set up my spring-boot HTTPS. I am generating a self-signed certificate during the building of my spring-boot application, using the command:

keytool
  -genkey
  -alias <alias>
  -dname <dname>
  -keyalg RSA
  -keysize 4096
  -storetype PKCS12
  -keystore <path_to_keystore>
  -validity 3650
  -storepass <keystore_pass>

I then export the cert to a .pem file, and extract the .crt and .key:

openssl pkcs12 -in cert.p12 -out cert.pem -nodes -passin pass:<pass>

This is mounted through a shared volume to my prometheus container, which has a --web.config.file containing:

tls_server_config:
  cert_file: /path/to/cert.crt
  key_file: /path/to/cert.key

And for good measure I added insecure_skip_verify: true to the prometheus.yml config:

- job_name: 'spring-actuator'
metrics_path: '/actuator/prometheus/'
scrape_interval: 60s
scheme: https
static_configs:
  - targets: [ '127.0.0.1:8443' ]
tls_config:
  insecure_skip_verify: true

Answer 1

Ok, I think I found my problem. I made two changes:

First, I moved the contents of the web.config.file into the prometheus.yml file under the 'spring-actuator'. Then I changed the target to use the hostname for my backend container, rather than 127.0.0.1.

The end result was a single prometheus.yml file:

- job_name: 'spring-actuator'
metrics_path: '/actuator/prometheus/'
scrape_interval: 60s
scheme: https
static_configs:
  - targets: [ 'backend:8443' ]
tls_config:
  cert_file: /path/to/cert.crt
  key_file: /path/to/cert.key
  insecure_skip_verify: true

So just some silly mistakes, not caused by the certs from what I can see. :)

Answer 2

Below is the critical part to debug the issue

target=https://127.0.0.1:8443/actuator/prometheus/
msg="Scrape failed"
err="Get \"https://127.0.0.1:8443/actuator/prometheus/\": dial tcp 127.0.0.1:8443: connect: connection refused"

It states that scrape has failed as it unable to connect to the target server. As the target server 127.0.0.1:8443 then it is expected to be in the same host where Prometheus is running.

Prometheus retrieval job, also called the scraper, pulls data from target services, aggregates it, and passes it to the database. Prometheus takes a list of scraping targets (IP addresses and ports) from a static list (or a file) called static_configs present in Prometheus YAML-based configuration file. More complex dynamic environments, where new instances might be brought up at any time, use service discovery mechanisms, which provide a list of machines to monitor and presents information of how these machines are organized.

When Prometheus scrapes a target, it attaches some labels automatically to the scraped time series which serve to identify the scraped target.

up{job="<job-name>", instance="<instance-id>"}: 1 if the instance is healthy, i.e. reachable, or 0 if the scrape failed.

job: The configured job name that the target belongs
instance: The : part of the target's URL that was scraped.

Configuring Prometheus instances

1. When Prometheus is running in localhost or same host.

scrape_configs:
- job_name: node
  static_configs:
  - targets: ['localhost:9100']

2. When Prometheus is running in different environment or host

scrape_configs: 
  - job_name: prometheus 
    static_configs: 
      - targets: ["localhost:9090"] 
  - job_name: eventservice 
    static_configs: 
      - targets: ["events:9090"] 
  - job_name: bookingservice 
    static_configs: 
      - targets: ["bookings:9090"] 

3. When running in multiple ports using IPs

static_configs:
  - targets: ['192.168.1.117:':8080', '192.168.1.117:8081']

TLS Config

TLS is used for establishing a secure, private transport between the Prometheus instance and scrape targets. By default, the Prometheus instance attempts to verify the certificate exposed by the scrape targets against its trust store, for establishing authenticity of the scrape targets.

scrape_configs:
  - job_name: 'node'
    scheme: https
    tls_config:
        # Prometheus will check that the node_exporter presents a certificate
        # signed by this ca.
        ca_file: 'ca.crt'
        # The cert and key are presented to node_exporter to authenticate
        # Prometheus as a client.
        cert_file: 'client.crt'
        key_file: 'client.key'

    static_configs:
    - targets: ['myserver.net:443']