Copying data between two google cloud projects and best practice for DEV & PROD environment in data centric environment

Question

I have two questions:

1. As storage bucket names are unique, how do I keep bucket name exactly same in development environment and production environment. Or what are best practice for dev and prod environment in data based environment?

2. How do i copy data from one project to other. I tried searching but i could not get efficient way to copy between 2 projects.

PS: Storage transfer allows copying between 2 buckets within same project, not cross project. I was not able to find bucket from different project even with search option. I searched using gs://another-project-bucket

Answer 1

First question

As storage bucket names are unique, how do I keep bucket name exactly same in development environment and production environment. Or what are best practice for dev and prod environment in data based environment?

You are correct. As far as Google Cloud Storage buckets is concerned, bucket names reside in a single Cloud Storage namespace. As per the documentation, this means that:

Every bucket name must be unique. Bucket names are publicly visible. If you try to create a bucket with a name that already belongs to an existing bucket, Cloud Storage responds with an error message. However, once you delete a bucket, you or another user can reuse its name for a new bucket.

As for best practices for development and production environment, I would say that the so-called "separation of concerns" would be the best option here. Having one single project for development purposes, and having a different project for production purposes separetly would be the best fit . Nonetheless, you can have both environments, env and prod, running within a single project; although, this option is not ideal in some cases.

---

Second question

How do i copy data from one project to other. I tried searching but i could not get efficient way to copy between 2 projects.

The answer can vary for this question:

1. You can copy GCS bucket objects across projects using the gsutil cp command, REST APIs, or GCS Client Libraries (Java, Node.js, Python). More info can be found here.
2. You can also achieve this using the Cloud Storage Data Transfer Service to move data from one Cloud Storage bucket to another, so that it is available to different groups of users or applications. Check the link for more information.

An example using gsutil cp would be as follows:

gsutil cp gs://[SOURCE_BUCKET_NAME]/[SOURCE_OBJECT_NAME] gs://[DESTINATION_BUCKET_NAME]/[NAME_OF_COPY]

Where:

[SOURCE_BUCKET_NAME] is the name of the bucket containing the object you want to copy. For example, my-bucket.

[SOURCE_OBJECT_NAME] is the name of the object you want to copy. For example, pets/dog.png.

[DESTINATION_BUCKET_NAME] is the name of the bucket where you want to copy your object. For example, another-bucket.

[NAME_OF_COPY] is the name you want to give the copy of your object. For example, shiba.png.

---

IMPORTANT: Make sure that you have the correct set permissions to perform this type of operation

You can also check How can I move data directly from one Google Cloud Storage project to another?.