Prometheus : Distinguish Application metrics

Question

I am running my services on EKS clusters. In order to collect the application metrics [API response times, status and number of calls], I came across Prometheus. There are following steps that I think needs to be done:

1. Cluster role, Service account and role binding: this will allow my prometheus service to talk to the cluster nods, pods and services [defined in the resources section].
2. Configmap: this allows the scraping process and defines different roles.
3. Service and ingress: to establish the endpoints [e.g.: 9090] and routes the traffic from internet.

I came across prometheus using helm which describes how we can make use of helm predefined prometheus charts in order to get the raw metrics from kubernetes.

I followed the steps:

kubectl create namespace prometheus

helm install prometheus stable/prometheus \
    --namespace prometheus \
    --set alertmanager.persistentVolume.storageClass="gp2",server.persistentVolume.storageClass="gp2"

kubectl get pods -n prometheus

I can see the pods running with that namespace. Now, I have two questions,

1. I am having multiple services (For example, service A and service B) running on the cluster. So, how can I distinguish the metrics on Prometheus.
2. Do I need to run kubectl --namespace=prometheus port-forward deploy/prometheus-server 9090 everytime to see the results? I see targetPort is defined as 9090 then why do I need to run the command? Can I just values.yaml instead?

Answer 1

Well to answer your questions

1. Yes, you can distinguish the service metrics by using label just use like this in your configMap of prometheus

    static_configs:
      - targets:
          - "<yourfirstservicename>.<namespace>.svc.cluster.local:<yourservice1portnumber>"
        labels:
          instance: 'service1'
      - targets:
          - "<yourservice2name>.<namespace>.svc.cluster.local:<yourservice2port>"
        labels:
          instance: 'service2'
   

2. Yes you have to do that port-forward but if you are planning to use grafana for visualization then new grafana version provide built in query run functionality.

I hope this will help !!