keda-operator-metrics-apiserver: no matching metrics found for s0-prometheus-http_requests_in_progress

Question

I am getting below error on keda-operator-metrics-apiserver pod logs when trying to scale using KEDA.

E0602 05:26:23.321583 1 status.go:71] apiserver received an error that is not an metav1.Status: &status.Error{s:(*status.Status)(0xc00105e2c0)}: rpc error: code = Unknown desc = error when getting metric values no matching metrics found for s0-prometheus-http_requests_in_progress

As we can see in this error, it says, no matching metrics found for s0-prometheus-http_requests_in_progress. It has automatically added s0-prometheus- to the metricName. What is the reason for that?

How can I solve this issue and read the metrics and scale my container?

I have created a KEDA ScaledObject as below.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: keda-so-prmts
  namespace: mynamespace
spec:
  scaleTargetRef:
    name: abc-test-yy
  pollingInterval:  10
  cooldownPeriod:   300
  minReplicaCount:  1
  maxReplicaCount:  10
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://10.1.254.12:9090
      metricName: http_requests_in_progress
      query: sum(http_requests_in_progress{action="getData", container="abc-test-yy", controller="Fetching", endpoint="8080", exported_endpoint="api/FetchService/getData/{productId}", job="abc-test-yy", method="POST", namespace="mynamespace", service="abc-test-yy"})by(container)
      threshold: '1'
      namespace: mynamespace

Answer 1

Had the same issue and finally this one was working for me:

• check the status.externalMetricNames inside the scaledObject resource that get assigned by KEDA once the object is created (those names match the metric names inside the hpa resource that gets created by KEDA)
• for prometheus based metrics you get s0-prometheus and s1-prometheus etc.
• seems like KEDA creates some kind of pseudo-CRD's with those names, because those queries work then
• kubectl get --raw /apis/external.metrics.k8s.io/v1beta1/namespaces/mynamespace/s0-prometheus?labelSelector=scaledobject.keda.sh%2Fname%3Dname-of-scaled-object and
• kubectl get --raw /apis/external.metrics.k8s.io/v1beta1/namespaces/mynamespace/s1-prometheus?labelSelector=scaledobject.keda.sh%2Fname%3Dname-of-scaled-object