Google App Engine Deploy has stopped working in 2023 - Cannot read property '@google-cloud/pubsub' of undefined

Question

I have an App Engine project running on Node 14. It's a Next.JS app that is built and then deployed via cloud build.

When I was committing code back in December 2022, it worked perfectly fine. However, this month (February 2023), it's suddenly not working. I haven't touched the project in months, haven't changed anything. I've also force-locked all my package.json deps (removing the caret/tilde symbols in version numbers), but I get the following error:

Already have image (with digest): gcr.io/cloud-builders/gcloud
Services to deploy:

descriptor:                  [/workspace/app.yaml]
source:                      [/workspace]
target project:              [foo]
target service:              [default]
target version:              [foo]
target url:                  [https://foo.nw.r.appspot.com]
target service account:      [App Engine default service account]


Do you want to continue (Y/n)?  
Beginning deployment of service [default]...
╔════════════════════════════════════════════════════════════╗
╠═ Uploading 73 files to Google Cloud Storage               ═╣
╚════════════════════════════════════════════════════════════╝
File upload done.
Updating service [default]...
......................................................................................................................................................................................................................................................................failed.
ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build (HASH HERE) status: FAILURE
npm ERR! Cannot read property '@google-cloud/pubsub' of undefined

My pubsub definition is like this:

import { PubSub } from '@google-cloud/pubsub';

My inkling would be that pubsub no longer allows these kinds of imports. But how is that even possible when I've locked the dependencies, and not touched anything in months?

And if not, any other ideas as to what could be causing it?

I have, of course, done the npm "turn it off and on again" with package lock and node_modules!

Also please note this only happens in the app engine deploy - the step in cloudbuild that compiles the code is fine.

Extra data from inside the cloudbuild's npm build part:

DEBUG: ***** CACHE MISS: "npm_modules"
Installing application dependencies.
--------------------------------------------------------------------------------
Running "node -v"
v14.21.2
Done "node -v" (6.244263ms)
--------------------------------------------------------------------------------
Running "npm --version"
6.14.17
Done "npm --version" (210.983956ms)
--------------------------------------------------------------------------------
Running "npm ci --quiet (NODE_ENV=production)"
npm ERR! Cannot read property '@google-cloud/pubsub' of undefined

cloudbuild.yaml

steps:
- name: node
  entrypoint: npm
  args: ['install']
- name: node
  entrypoint: npm
  args: ['run', 'build']
- name: "gcr.io/cloud-builders/gcloud"
  args: ["app", "deploy"]

timeout: "1600s"

app.yaml

runtime: nodejs14
service: default
instance_class: F2
handlers:
  - url: /.*
    secure: always
    script: auto

Gcloud version will be whatever gcr.io/cloud-builders/gcloud uses, I assume latest. My local is:

Google Cloud SDK 412.0.0
alpha 2022.12.09
beta 2022.12.09
bq 2.0.83
core 2022.12.09
gsutil 5.17
Updates are available...

The important thing to note is that this used to work a few months ago, and now it does not - with no configuration changes

Update:

This github issue also suggests that it could be an issue with the caching layer on google cloudbuild, which would explain the mystery lack of the first module it encounters (same issue on multiple projects) - it can't find the node_modules/ -- if it ends up being that then I'll close this question. https://github.com/GoogleCloudPlatform/buildpacks/issues/233#issuecomment-1452453245

package.json

{
  "name": "foo",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "DEBUG=foo:* next dev",
    "build": "next build",
    "start": "DEBUG=foo:* next start",
    "lint": "next lint",
    "test:cypress": "cypress run"
  },
  "dependencies": {
    "@google-cloud/pubsub": "3.3.0",
    "@google-cloud/secret-manager": "4.1.2",
    "@google-cloud/storage": "6.4.2",
    "@googlemaps/google-maps-services-js": "3.3.16",
    "critters": "0.0.7",
    "debug": "4.3.4",
    "firebase-admin": "11.0.1",
    "handlebars": "4.7.7",
    "handlebars-loader": "1.7.2",
    "moment": "2.29.4",
    "moment-timezone": "0.5.37",
    "next": "12.2.2",
    "nodemailer": "6.7.8",
    "puppeteer": "17.1.3",
    "react": "18.2.0",
    "react-dom": "18.2.0",
    "soap": "0.45.0",
    "styled-components": "5.3.5",
    "use-persisted-state": "0.3.3",
    "uuid": "9.0.0",
    "xml2js": "0.4.23"
  },
  "devDependencies": {
    "@babel/core": "7.18.6",
    "@babel/eslint-parser": "7.18.2",
    "babel-eslint": "10.1.0",
    "cypress": "10.7.0",
    "eslint": "8.19.0",
    "eslint-config-airbnb": "19.0.4",
    "eslint-config-next": "12.2.2",
    "eslint-import-resolver-jsconfig": "1.1.0",
    "eslint-plugin-import": "2.26.0",
    "eslint-plugin-jsx-a11y": "6.6.0",
    "eslint-plugin-react": "7.30.1",
    "eslint-plugin-react-hooks": "4.6.0"
  }
}

Answer 1

It seems to be a problem with node v14. I replicated the problem with v14, but didn't face the problem with v12 and v16.

could you experiment with that?