Builder Permission denied. Unable to deploy on Google cloud

Question

ERROR: build step 1 "gcr.io/gae-runtimes/nodejs8_app_builder:nodejs8_20180618_RC02" failed: exit status 1
ERROR
Finished Step #1 - "builder"
Step #1 - "builder": Permission denied for "be8392bdf4a2c92301391a124a5b72078453db3c15fcfc71f923e3c63d1bd8ea" from request "/v2/PROJECT_ID/app-engine-build-cache/node-cache/manifests/be8392bdf4a2c92301391a124a5b72078453db3c15fcfc71f923e3c63d1bd8ea". : None
Step #1 - "builder": containerregistry.client.v2_2.docker_http_.V2DiagnosticException: response: {'status': '403', 'content-length': '291', 'x-xss-protection': '1; mode=block', 'transfer-encoding': 'chunked', 'server': 'Docker Registry', '-content-encoding': 'gzip', 'docker-distribution-api-version': 'registry/2.0', 'cache-control': 'private', 'date': 'Sun, 15 Jul 2018 08:26:14 GMT', 'x-frame-options': 'SAMEORIGIN', 'content-type': 'application/json'}
Step #1 - "builder": File "/ftl-v0.4.0.par/containerregistry/client/v2_2/docker_http_.py", line 364, in Request
Step #1 - "builder": File "/ftl-v0.4.0.par/containerregistry/client/v2_2/docker_image_.py", line 250, in _content
Step #1 - "builder": File "/ftl-v0.4.0.par/containerregistry/client/v2_2/docker_image_.py", line 293, in manifest
Step #1 - "builder": File "/ftl-v0.4.0.par/containerregistry/client/v2_2/docker_image_.py", line 279, in exists
Step #1 - "builder": File "/ftl-v0.4.0.par/__main__/ftl/common/cache.py", line 166, in getEntryFromCreds
Step #1 - "builder": File "/ftl-v0.4.0.par/__main__/ftl/common/cache.py", line 143, in _getLocalEntry
Step #1 - "builder": File "/ftl-v0.4.0.par/__main__/ftl/common/cache.py", line 128, in _getEntry
Step #1 - "builder": File "/ftl-v0.4.0.par/__main__/ftl/common/cache.py", line 110, in Get
Step #1 - "builder": File "/ftl-v0.4.0.par/__main__/ftl/node/layer_builder.py", line 55, in BuildLayer
Step #1 - "builder": File "/ftl-v0.4.0.par/__main__/ftl/node/builder.py", line 38, in Build
Step #1 - "builder": File "/ftl-v0.4.0.par/__main__.py", line 52, in main
Step #1 - "builder": File "/ftl-v0.4.0.par/__main__.py", line 61, in 
Step #1 - "builder": exec code in run_globals
Step #1 - "builder": File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
Step #1 - "builder": "__main__", fname, loader, pkg_name)
Step #1 - "builder": File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
Step #1 - "builder": Traceback (most recent call last):
Step #1 - "builder": INFO full build took 0 seconds
Step #1 - "builder": INFO build process for FTL image took 0 seconds
Step #1 - "builder": INFO checking_cached_packages_json_layer took 0 seconds
Step #1 - "builder": DEBUG Checking cache for cache_key be8392bdf4a2c92301391a124a5b72078453db3c15fcfc71f923e3c63d1bd8ea
Step #1 - "builder": INFO starting: checking_cached_packages_json_layer
Step #1 - "builder": INFO starting: build process for FTL image
Step #1 - "builder": INFO builder initialization took 0 seconds
Step #1 - "builder": INFO Loading Docker credentials for repository 'asia.gcr.io/PROJECT_ID/app-engine/default/20180715t135547:6382e88f-f9db-4087-9eca-2a1aee5881d6'
Step #1 - "builder": INFO Loading Docker credentials for repository 'gcr.io/gae-runtimes/nodejs8:nodejs8_20180618_RC02'
Step #1 - "builder": INFO starting: builder initialization
Step #1 - "builder": INFO starting: full build
Step #1 - "builder": INFO FTL arg passed: verbosity NOTSET
Step #1 - "builder": INFO FTL arg passed: destination_path /srv
Step #1 - "builder": INFO FTL arg passed: entrypoint None
Step #1 - "builder": INFO FTL arg passed: directory /workspace
Step #1 - "builder": INFO FTL arg passed: output_path None
Step #1 - "builder": INFO FTL arg passed: base gcr.io/gae-runtimes/nodejs8:nodejs8_20180618_RC02
Step #1 - "builder": INFO FTL arg passed: upload True
Step #1 - "builder": INFO FTL arg passed: cache True
Step #1 - "builder": INFO FTL arg passed: global_cache False
Step #1 - "builder": INFO FTL arg passed: name asia.gcr.io/PROJECT_ID/app-engine/default/20180715t135547:6382e88f-f9db-4087-9eca-2a1aee5881d6
Step #1 - "builder": INFO FTL arg passed: builder_output_path /builder/outputs
Step #1 - "builder": INFO FTL arg passed: tar_base_image_path None
Step #1 - "builder": INFO FTL arg passed: cache_repository asia.gcr.io/PROJECT_ID/app-engine-build-cache
Step #1 - "builder": INFO FTL arg passed: exposed_ports None
Step #1 - "builder": INFO Beginning FTL build for node
Step #1 - "builder": INFO FTL version node-v0.4.0
Step #1 - "builder": Status: Downloaded newer image for gcr.io/gae-runtimes/nodejs8_app_builder:nodejs8_20180618_RC02
Step #1 - "builder": Digest: sha256:f937017daa12ccde31d70836e640ef0eaf327436695d05da38e4290c2eb2eb70
Step #1 - "builder": nodejs8_20180618_RC02: Pulling from gae-runtimes/nodejs8_app_builder
Step #1 - "builder": Pulling image: gcr.io/gae-runtimes/nodejs8_app_builder:nodejs8_20180618_RC02
Starting Step #1 - "builder"
Finished Step #0 - "fetcher"
Step #0 - "fetcher": 2018/07/15 08:26:12 ******************************************************
Step #0 - "fetcher": 2018/07/15 08:26:12 Total time: 2.04 s
Step #0 - "fetcher": 2018/07/15 08:26:12 Time for manifest: 888.14 ms
Step #0 - "fetcher": 2018/07/15 08:26:12 MiB/s throughput: 0.28 MiB/s
Step #0 - "fetcher": 2018/07/15 08:26:12 MiB downloaded: 0.33 MiB
Step #0 - "fetcher": 2018/07/15 08:26:12 GCS timeouts: 0
Step #0 - "fetcher": 2018/07/15 08:26:12 Total retries: 0
Step #0 - "fetcher": 2018/07/15 08:26:12 Total files: 16
Step #0 - "fetcher": 2018/07/15 08:26:12 Actual workers: 16
Step #0 - "fetcher": 2018/07/15 08:26:12 Requested workers: 200
Step #0 - "fetcher": 2018/07/15 08:26:12 Completed: 2018-07-15T08:26:12Z
Step #0 - "fetcher": 2018/07/15 08:26:12 Started: 2018-07-15T08:26:10Z
Step #0 - "fetcher": 2018/07/15 08:26:12 Status: SUCCESS
Step #0 - "fetcher": 2018/07/15 08:26:12 ******************************************************
Step #0 - "fetcher": 2018/07/15 08:26:12 Fetched gs://staging.PROJECT_ID.appspot.com/5e9210875e15a2eb7d50c666136266837638eb03 (322831B in 1.145458029s, 0.27MiB/s)
Step #0 - "fetcher": 2018/07/15 08:26:12 Fetched gs://staging.PROJECT_ID.appspot.com/058b7e502bf6750c6f01453ef947d5dd7e854e07 (1279B in 861.521735ms, 0.00MiB/s)
Step #0 - "fetcher": 2018/07/15 08:26:12 Fetched gs://staging.PROJECT_ID.appspot.com/5d87534d139519ba7cec4d48d2c3ba27b99e80b0 (319B in 846.233597ms, 0.00MiB/s)
Step #0 - "fetcher": 2018/07/15 08:26:12 Fetched gs://staging.PROJECT_ID.appspot.com/668461d157d199b783be12fd2f2ba9c6d154130c (1271B in 838.8342ms, 0.00MiB/s)
Step #0 - "fetcher": 2018/07/15 08:26:11 Fetched gs://staging.PROJECT_ID.appspot.com/8dd0bca621558e6ce972f38d8aa9765e49436172 (327B in 589.603096ms, 0.00MiB/s)
Step #0 - "fetcher": 2018/07/15 08:26:11 Fetched gs://staging.PROJECT_ID.appspot.com/c6b4f62664bbe2779fdff10261bc384708e73e7d (36B in 588.36617ms, 0.00MiB/s)
Step #0 - "fetcher": 2018/07/15 08:26:11 Fetched gs://staging.PROJECT_ID.appspot.com/c271c61f49150008a5eeae99a9bd09570bd5d549 (2558B in 588.802856ms, 0.00MiB/s)
Step #0 - "fetcher": 2018/07/15 08:26:11 Fetched gs://staging.PROJECT_ID.appspot.com/f293a6701e6de41513ba08b50bbbacb58ffbc19a (119B in 586.215815ms, 0.00MiB/s)
Step #0 - "fetcher": 2018/07/15 08:26:11 Fetched gs://staging.PROJECT_ID.appspot.com/88ef0fc94347f5e38160bdacef44fedc51b85877 (2305B in 584.714922ms, 0.00MiB/s)
Step #0 - "fetcher": 2018/07/15 08:26:11 Fetched gs://staging.PROJECT_ID.appspot.com/73c3a01c14c7b696454020d6dc917ea32a50872c (53B in 584.291891ms, 0.00MiB/s)
Step #0 - "fetcher": 2018/07/15 08:26:11 Fetched gs://staging.PROJECT_ID.appspot.com/757cebb200f220928061cdabddd6126d67a46984 (1683B in 583.3675ms, 0.00MiB/s)
Step #0 - "fetcher": 2018/07/15 08:26:11 Fetched gs://staging.PROJECT_ID.appspot.com/1bbf5d56ff53e82b6444cab2a87d43b14d23a8b3 (217B in 584.334096ms, 0.00MiB/s)
Step #0 - "fetcher": 2018/07/15 08:26:11 Fetched gs://staging.PROJECT_ID.appspot.com/42a994f47562d763d59a4b64822554d24f0ffce7 (28B in 577.748008ms, 0.00MiB/s)
Step #0 - "fetcher": 2018/07/15 08:26:11 Fetched gs://staging.PROJECT_ID.appspot.com/f9185bdfc5b57bef24b52a9bbeb09cdf981f279f (2495B in 579.458821ms, 0.00MiB/s)
Step #0 - "fetcher": 2018/07/15 08:26:11 Fetched gs://staging.PROJECT_ID.appspot.com/0c676a042d5ad4e493046f23965ed648293225be (3162B in 577.802151ms, 0.01MiB/s)
Step #0 - "fetcher": 2018/07/15 08:26:11 Fetched gs://staging.PROJECT_ID.appspot.com/5bf6fe209f6ec1a459ae628d8f94e6aedbf3abae (2675B in 571.413697ms, 0.00MiB/s)
Step #0 - "fetcher": 2018/07/15 08:26:11 Processing 16 files.
Step #0 - "fetcher": 2018/07/15 08:26:11 Fetched gs://staging.PROJECT_ID.appspot.com/ae/6382e88f-f9db-4087-9eca-2a1aee5881d6/manifest.json (3576B in 888.138003ms, 0.00MiB/s)
Step #0 - "fetcher": 2018/07/15 08:26:10 Fetching manifest gs://staging.PROJECT_ID.appspot.com/ae/6382e88f-f9db-4087-9eca-2a1aee5881d6/manifest.json.
Step #0 - "fetcher": Already have image (with digest): gcr.io/cloud-builders/gcs-fetcher
Starting Step #0 - "fetcher"
BUILD
FETCHSOURCE

starting build "88e0cc7f-62b7-496a-aa31-230e793b5ea1"

Answer 1

Troubleshooting

If you find 403 (access denied) errors in your build logs, try the following steps:

Disable the Cloud Build API and re-enable it. Doing so should give your service account access to your project again.

Fixed an issue for me.

Answer 2

Its a weird error, because it just says permission denied in logs.

In my case Actual reason was unpaid bills, which keeps Cloud Build API enabled just for suspense but actually deployment doesn't work. So do these steps

1. Pay your bills if there is any pending, in short your Payment Method has to be active in Billings for the particular project you are getting error

2. After bill is paid, disable the Cloud Build API and re-enable it, wait for few seconds/minutes and deploy again.