Prometheus Helm Chart: Scrape Target not detected with prometheus.io/scrape annotations [closed]

Question

I'm using this prometheus chart. In the documentation it says

In order to get prometheus to scrape pods, you must add annotations to the the pods as below:

metadata:   
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/path: /metrics
    prometheus.io/port: "8080"

So I have created a service like this

apiVersion: v1
kind: Service
metadata:
  name: nodejs-client-service
  labels:
    app: nodejs-client-app
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/path: /metrics
    prometheus.io/port: "5000"
spec:
  type: LoadBalancer
  selector:
    app: nodejs-client-app
  ports:
    - protocol: TCP
      name: http
      port: 80
      targetPort: 5000

But my Service won't show up in the prometheus targets. What am I missing?

Answer 1

I ran into the same problem with the stable/prometheus-operator chart. I tried adding the annotations above both to the pods and to the service and neither worked.

For me, the solution was to add a ServiceMonitor object. Once added, Prometheus dynamically discovered my service:

Fig 1: target list

Fig 2: dynamically added scrape_config

Solution Example

This single command fixed the problem: kubectl apply -f service-monitor.yml

# service-monitor.yml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    release: prom
  name: eztype
  namespace: default
spec:
  endpoints:
    - path: /actuator/prometheus
      port: management
  namespaceSelector:
    matchNames:
      - default
  selector:
    matchLabels:
      app.kubernetes.io/name: eztype

Here, my pods and service were annotated with the name eztype and were exposing metrics on port 8282 under the indicated path. For completeness here's the relavant part of my service definition:

# service definition (partial)
spec:
  clusterIP: 10.128.156.246
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  - name: management
    port: 8282
    protocol: TCP
    targetPort: 8282

It's worth noting that ServiceMonitor objects are used in the Prometheus chart itself:

$ kubectl get servicemonitors -n monitor
NAME                                               AGE
prom-prometheus-operator-alertmanager              14d
prom-prometheus-operator-apiserver                 14d
prom-prometheus-operator-coredns                   14d
prom-prometheus-operator-grafana                   14d
prom-prometheus-operator-kube-controller-manager   14d
prom-prometheus-operator-kube-etcd                 14d
prom-prometheus-operator-kube-proxy                14d
prom-prometheus-operator-kube-scheduler            14d
prom-prometheus-operator-kube-state-metrics        14d
prom-prometheus-operator-kubelet                   14d
prom-prometheus-operator-node-exporter             14d
prom-prometheus-operator-operator                  14d
prom-prometheus-operator-prometheus                14d