Configuring Prometheus for Spring Boot health check monitoring

Question

We have few Spring Boot applications that have health check implemented. The response for those checks was changed to JSON format based on @Thiru's suggestion. I now get the following response:

Prometheus server is running on Ubuntu instance. Spring boot services that have to be monitored are running on Windows server 2016. I have installed blackbox-exporter (version 0.12.0.windows-amd64) on Windows server after seeing this post.

Following change was done on client (Windows server with IP: 172.16.x.yz) side in blackbox.yml:

modules:
  http_2xx:
    prober: http
    http:
  http_post_2xx:
    prober: http
    timeout: 5s
    http:
      method: POST
      headers:
        Content-Type: application/json
      body: '{"status": "UP"}'
  ...
  ...

On Prometheus server, following is the content of prometheus.yml:

...
...
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      #module: [http_2xx]  # Look for a HTTP 200 response.
      module: [http_post_2xx]  # Look for a HTTP 200 response.
    static_configs:
      - targets:
        - http://172.16.x.yz:6300/serviceA/actuator/health
        - http://172.16.x.yz:6340/serviceB/actuator/health
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 172.16.x.yz:9115  # The blackbox exporter's real hostname:port.

After making above changes on client and server, when i restart both the blackbox-exporter and prometheus, i see that Prometheus always shows State as UP even if the two services that the exporter is monitoring go down. Seems Prometheus is displaying the status of the blackbox-exporter and not for the services. Any suggestion how i can fix that?

Answer 1

I would recommend to use some kind of tools which helps in exporting the spring health as prometheus exporter. Basically, exporter which helps in converting json data from an http url into prometheus metrics using jsonpath like:

• https://github.com/project-sunbird/prometheus-jsonpath-exporter
• https://github.com/thirunar/prometheus-spring-boot-info-exporter