When creating a new cloud composer env, is it possible to set the bucket to preexisting one?

Question

So I already have an empty storage bucket created for this and I don't want composer to create its own bucket for the dags - I'd like to use the one already created.

It's not ideal to have it just create a random bucket and then go

gcloud composer environments run test-environment  --location europe-west1 variables -- --set gcs_bucket gs://my-bucket

I've dug around the docs but it seems you cannot go around it creating a brand new bucket every time?

Answer 1

Currently, it is not possible.

In the environment’s configuration in Cloud Composer API, the dagGcsPrefix parameter is output only, you cannot set it. Documentation also mentions a Cloud Storage bucket is always created along with the Composer environment, the name of the bucket is based on the environment’s region, name and a random Id.

You may want to “Star” this Feature Request for the mentioned functionality, to receive notifications whenever an update on this regard is published. You can also review or subscribe to the Cloud Composer release notes to be updated about recently added features.

Answer 2

You are right, this is currently not supported in Composer.