How to generate signed URLs for Google Cloud Storage objects in GKE (Go)

Question

Goal:

Generate signed URLs inside GKE pods without manually injecting a service account JSON key. The syntax for generating them requires a service account email and private key.

//import "cloud.google.com/go/storage"
url, err := storage.SignedURL(bucketName, objectName, &storage.SignedURLOptions{
    ContentType:    contentType,
    GoogleAccessID: saEmail,
    PrivateKey:     saPrivateKey,
})

In other words, I'd like to load saEmail and saPrivateKey from the default credentials automatically available in GKE nodes.

Attempt:

ctx := context.Background()
//errors ignored for brevity
//import "golang.org/x/oauth2/google"
creds, _ := google.FindDefaultCredentials(ctx, storage.ScopeReadWrite)
cfg, _ := google.JWTConfigFromJSON(creds.JSON)
url, _ := storage.SignedURL(bucketName, objectName, &storage.SignedURLOptions{
    ContentType:    contentType,
    GoogleAccessID: cfg.Email,
    PrivateKey:     cfg.PrivateKey,
})

When I ran google.FindDefaultCredentials() inside a GKE pod, the result JSON is empty.

Environment:

• Go 1.13
• GKE 1.14.10-gke.36
• cloud.google.com/go v0.58.0
• cloud.google.com/go/storage v1.8.0

Additional Notes:

I've tested two possible alternatives involving injecting the service account key (JSON) manually into the pod, but I hope to avoid them if possible:

• Writing the service account key into a file and setting GOOGLE_APPLICATION_CREDENTIALS to its path. When this is done, google.FindDefaultCredentials() loads the email and private key.

• Passing the service account key as a string into the pod and parsing it with google.JWTConfigFromJSON().

Answer 1

For generating a signed URL, you need to have a private key.

When you are on GCP services (here on Compute Instances, the node of your K8S cluster, but it's the same thing with Cloud RUn, Cloud Functions et other GCP services) and you use the default credential (and there is no GOOGLE_APPLICATION_CREDENTIALS env var defined), the library use the metadata server.

The metadata server allows you to generate an access token

curl -H "Metadata-Flavor: Google" \
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token

or an identity token (with the audience in parameter)

curl -H "Metadata-Flavor: Google" \
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/identity?audience=https://www.google.com

Thus, without having any secret or private key on your side, the libraries are able to generate a token (access or identity) for reaching external API.

However, the metadata server doesn't provide the secret (the private key) and you can't use it to generated signed URLs.

You need a service account key file here

You have several way to provide it to the pod in secure manner.

• The standard K8S way: use a secret volume
• The GCP managed solution: Secret manager. Thanks to the metadata server, you can instantiate the secret manager client and get your secret (your service account key file) and then use it as before. An alternative, I wrote a script that preload the secret into env variables when the container start, and thus, you just have to use the env var without interacting with the secret manager service

I don't recommend you to put your service account key file directly in the container, it's not really secure

Another solution

Ultimately, you can generate on the fly a key and defined it as the service account key (it's name user-defined service account key).

• You could generate it when you deploy your service on the cluster. Like this, you don't have to store a secret, it's generated on the fly each time.
• You could generate a key at container startup, set it in the service account, and keep it in memory.

Then, use it when you need it in your code and it should work because it's link to your service account.

However, you also need to think how to clean the old and useless keys.