Configure Prometheus alerting rules combining the status of 2 different instances

Question

I'm trying to configure into Prometheus Alerting Manager an alert that appears when the status of 2 different hosts is down. To better explain, I have these couples of hosts (host=instance):

host1.1
host1.2

host2.1
host2.2

host3.1 
host3.2

host4.1
host4.2
...

and I need an alert that appears when both hosts of the SAME couple are DOWN:

expr = ( icmpping{instance=~"hostX1"}==0 and icmpping{instance=~"hostX2"}==0 )

(I know that the syntax is not correct, I just wanted to underline that X refers to the same number on both icmppingconditions)

Any hint?

Answer 1

The easiest way is perhaps to generate a label at ingestion time reflecting this logic, using relabel_config

relabel_configs:
  - source_labels: [host]
    regex: ^(.*)\.\d+$
    target_label: host_group

It will generate the label you need for matching:

host=host1.1 => host_group=host1
host=host1.2 => host_group=host1

You can then use it for your alerting rules.

sum(icmpping) on(host_group) == 0

If this is not possible, you can use label_replace to achieve the same (only on instant vectors)

sum(label_replace(icmpping,"host_group","$1","host","(.*)\._\\d+")) on(host_group) == 0