Prometheus for k8s multi clusters

Question

I have 3 kubernetes clusters (prod, test, monitoring). Iam new to prometheus so i have tested it by installing it in my test environment with the helm chart:

# https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
helm install [RELEASE_NAME] prometheus-community/kube-prometheus-stack

But if i want to have metrics from the prod and test clusters, i have to repeat the same installation of the helm and each "kube-prometheus-stack" would be standalone in its own cluster. It is not ideal at all. Iam trying to find a way to have a single prometheus/grafana which would federate/agregate the metrics from each cluster's prometheus server.

I found this link, saying about prometheus federation:

https://prometheus.io/docs/prometheus/latest/federation/

If install the helm chart "kube-prometheus-stack" and get rid of grafana on the 2 other cluster, how can i make the 3rd "kube-prometheus-stack", on the 3rd cluster, scrapes metrics from the 2 other ones?
thanks

Answer 1

You have to modify configuration for prometheus federate so it can scrape metrics from other clusters as described in documentation:

scrape_configs:
  - job_name: 'federate'
    scrape_interval: 15s

    honor_labels: true
    metrics_path: '/federate'

    params:
      'match[]':
        - '{job="prometheus"}'
        - '{__name__=~"job:.*"}'

    static_configs:
      - targets:
        - 'source-prometheus-1:9090'
        - 'source-prometheus-2:9090'
        - 'source-prometheus-3:9090'

params field checks for jobs to scrape metrics from. In this particular example

It will scrape any series with the label job="prometheus" or a metric name starting with job: from the Prometheus servers at source-prometheus-{1,2,3}:9090

You can check following articles to give you more insight of prometheus federation:

1. Monitoring Kubernetes with Prometheus - outside the cluster!

2. Prometheus federation in Kubernetes

3. Monitoring multiple federated clusters with Prometheus - the secure way

4. Monitoring a Multi-Cluster Environment Using Prometheus Federation and Grafana