Configuring HPA using cross project pub/sub metrics in GCP

Question

I am trying to configure HPA for GKE workload in one of our projects (lets say ProjectA) using the pubsub metrics from another project of ours (lets say ProjectB). I receive the following permission denied error:

Warning FailedGetExternalMetric 9s (x12 over 2m56s) horizontal-pod-autoscaler unable to get external metric default/pubsub.googleapis.com|subscription|num_undelivered_messages/&LabelSelector{MatchLabels:map[string]string{resource.labels.project_id: ProjectB,resource.labels.subscription_id: ProjectB.Subscription,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server could not find the descriptor for metric pubsub.googleapis.com/subscription/num_undelivered_messages: googleapi: Error 403: Permission monitoring.metricDescriptors.get denied (or the resource may not exist)., forbidden

I have Mpnitoring.admin role assigned on both the projects and have pub/sub APIs enabled on both the projects.

Also, I am able to configure HPA for GKE workload in ProjectA using the pubsub metrics from with the ProjectA. I want to not extend the same configuration using metrics from ProjectB.

Can anybody assist me on the issue please?

Followed the KB to configure the HPA:

https://cloud.google.com/kubernetes-engine/docs/tutorials/autoscaling-metrics?_ga=2.189193512.-2038413145.1679461982#pubsub_7

Answer 1

You need to create a service account for custom-metrics-adapter workload in Project A, assign permissions to this IAM service account in Project B, and bind this IAM service account to custom-metrics Kubernetes Service account.

The given steps are for the GKE cluster with Workload Identity.

1. Create a IAM service account in Project A.

2. Assign Monitor Viewer role to this service account in Project B.

3. Mark IAM service account to be used as Kubernetes workload identity

   gcloud iam service-accounts add-iam-policy-binding --role
   roles/iam.workloadIdentityUser --member
   "serviceAccount:.svc.id.goog[custom-metrics/custom-metrics-stackdriver-adapter]"
   @.iam.gserviceaccount.com

4. Change custom-metrics-stackdriver-adapter Kubernetes Service account to use created IAM service account:

   kubectl annotate serviceaccount --namespace custom-metrics
   custom-metrics-stackdriver-adapter
   iam.gke.io/[email protected]

see details: Start Custom Metrics - Stackdriver Adapter section