Prometheus: filter query based on another metric

Question

Say I have two metrics in prometheus, both counters:

• requests_processed_total
• requests_failed_total

They both have a matching service label. Example:

requests_processed_total{service="news"} 1097
requests_processed_total{service="store"} 487
requests_failed_total{service="news"} 23
requests_failed_total{service="store"} 89

How to query the requests_failed_total, but only for services whose request_processed_total > 1000.

I'm expecting the following response:

requests_failed_total{service="news"} 23

# Note that the "store" service is excluded

Answer 1

requests_failed_total and on(service) requests_processed_total > 1000

https://prometheus.io/docs/prometheus/latest/querying/operators/#logical-set-binary-operators https://prometheus.io/docs/prometheus/latest/querying/operators/#one-to-one-vector-matches