Sops unable to gcp kms decrypt file on Circleci despite GOOGLE_APPLICATION_CREDENTIALS successfully set to service account json

Question

I am trying to configure a job on my local circleci (using docker executor, image: google/cloud-sdk:latest), and that job requires a sops gcp kms encrypted file to be decrypted. I have setup a google service account for the gcp kms decrypt service (I can run the script, to be run via the circleci job, successfully locally by decrypting the sops file via the service account, so I know the service account setup is valid). Here is how I am running my job.

1- I base64 encode the google service account json file: base64 path/to/service_aacount_file.json

2- I run circleci job, setting GCLOUD_SERVICE_KEY environment variable on circleci, with the base64 encoded content from the previous step: circleci local execute --env GCLOUD_SERVICE_KEY='<Base64EncodedServiceAccountJsonFileContent>' --job '<MyJob>'

3- Here is my circleci config:

- run:
          name: <MyJob>
          command: |
            apt-get install -y docker
            apt-get install -y sudo
            cd $(pwd)/path/to/jobcode
            echo $GCLOUD_SERVICE_KEY | base64 -d > ${HOME}/<MyGoogleServiceAccountJsonFile.json>
            export GOOGLE_APPLICATION_CREDENTIALS="${HOME}/<MyGoogleServiceAccountJsonFile.json>" 
            gcloud auth activate-service-account --key-file ${HOME}/<MyGoogleServiceAccountJsonFile.json>
            echo $GOOGLE_APPLICATION_CREDENTIALS
            ls -halt $GOOGLE_APPLICATION_CREDENTIALS
            cat $GOOGLE_APPLICATION_CREDENTIALS
            sudo ./<RunJob.sh>

4- I get following error when I execute the job:

Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
  projects/<MyProject>/locations/<MyLocation>/keyRings/<MySopsKeyring>/cryptoKeys/<MyKey>: FAILED
    - | Cannot create GCP KMS service: google: could not find
      | default credentials. See
      | https://developers.google.com/accounts/docs/application-default-credentials
      | for more information.

Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.

5- Further, from the console output:

a- I can see that the service account was successfully activated: Activated service account credentials for: [<MyServiceAccount>@<MyProject>.iam.gserviceaccount.com]

b- The GOOGLE_APPLICATION_CREDENTIALS environment variable is set to the service account json's path: /path/to/service_account.json

c- The above file has been correctly base64 decoded and contains valid json:

{
    "client_x509_cert_url": "<MyUrl>",
    "auth_uri": "<MyAuthUri>",
    "private_key": "<MyPrivateKey>",
    "client_email": "<ClientEmail>",
    "private_key_id": "<PrivateKeyId>",
    "client_id": "<ClientId>",
    "token_uri": "<TokenUri>",
    "project_id": "<ProjectId>",
    "type": "<ServiceAccount>",
    "auth_provider_x509_cert_url": "<AuthProviderCertUrl>"
}

6- Some other things I have tried:

a- Tried setting google project name in environment variables, but still same error.

b- Tried setting GOOGLE_APPLICATION_CREDENTIALS to file's content, instead of file path, but again same result.

c- Tried setting GOOGLE_APPLICATION_CREDENTIALS by providing file path without quotes or single quotes, but still no difference.

d- Tried setting $BASH_ENV by doing echo 'export GOOGLE_APPLICATION_CREDENTIALS=path/to/service_account.json' >> $BASH_ENV, but same error

Please help.

Answer 1

Five options that could work:

1. Try to run the following command: gcloud auth application-default login
2. Try this command to set the env var: echo 'export GOOGLE_APPLICATION_CREDENTIALS=/tmp/service-account.json' >> $BASH_ENV
3. The other thing is that I see that runjob.sh is running under root. It could be that the gcp credentials are not visible under sudo per default. Either run the script without sudo or run the preceding commands with sudo.
4. As a last resort (those options worked for me, could be different in your scenario): { echo 1; echo 1; echo n; } | gcloud init
5. gcloud components update This sometimes works when the sdk is outdated.
6. config set project [PROJECT_NAME]

You can also check active accounts with: gcloud auth list