Error migrating Google Cloud Container Registry to Artifact Registry: RESOURCE_EXHAUSTED

Question

We are following Google's instructions on migrating Container Registry to Artifact Registry for several App Engine Standard services that we have. We don't want to do this, but we have to, as Container Registry is shutting down.

Per Google's guide we are executing the command, gcloud artifacts docker upgrade migrate --projects=projectid

It succeeded for one project.

When we went to do the next project, it started failing with:

ERROR: (gcloud.artifacts.docker.upgrade.migrate) RESOURCE_EXHAUSTED: Resource has been exhausted (e.g. check quota).

I waited several hours and tried again but it still fails with the same error. So, I went to the quotas page and checked: we have not exceeded any quotas. I checked both the project quotas and the organization's quotas. Nada.

What can I do to fix this error so we can continue the migration? We have many services to migrate and this is NOT a good experience for migrating production services

Here's the http call that is failing with a 429 status code (over quota):

==== request start ====
uri: https://cloudasset.googleapis.com/v1/organizations/xxxx:analyzeIamPolicy?alt=json&analysisQuery.accessSelector.permissions=storage.objects.get&analysisQuery.accessSelector.permissions=storage.objects.list&analysisQuery.accessSelector.permissions=storage.objects.create&analysisQuery.accessSelector.permissions=storage.objects.delete&analysisQuery.resourceSelector.fullResourceName=%2F%2Fcloudresourcemanager.googleapis.com%2Fprojects%2Fxxxxx
method: GET

Answer 1

I am pretty sure the issue comes from the Policy Analyzer free tier limitation.

By default the Policy Analyzer is limited to 20 Queries per day: https://cloud.google.com/policy-intelligence/docs/policy-analyzer-overview#pricing

And if you look at the logs of the migration command it use the policy analyzer to validate some of your policy on the repos:

DEBUG: https://cloudasset.googleapis.com:443 "GET /v1/projects/<PROJECT>:analyzeIamPolicy?alt=json&analysisQuery.accessSelector.permissions=storage.objects.get&analysisQuery.accessSelector.permissions=storage.objects.list&analysisQuery.accessSelector.permissions=storage.objects.create&analysisQuery.accessSelector.permissions=storage.objects.delete&analysisQuery.resourceSelector.fullResourceName=%2F%2Fstorage.googleapis.com%2Fartifacts.<PROJECT>.appspot.com HTTP/1.1" 429 None

Using this error log you can recreate the query in the Policy Analyzer and you get the error message:

Upgrade to run analysis
You can run 20 queries per day
Try again tomorrow or upgrade to Security Comand Center Premium for unlimited queries.