How to upload multiple files to google cloud storage bucket as a transaction

Question

Use Case:

Upload multiple files into a cloud storage bucket, and then use that data as a source to a bigquery import. Use the name of the bucket as the metadata to drive which sharded table the data should go into.

Question:

In order to prevent partial import to the bigquery table, ideally, I would like to do the following,

• Upload the files into a staging bucket
• Verify all files have been uploaded correctly
• Rename the staging bucket to its final name (for example, gs://20130112)
• Trigger the bigquery import to load the bucket into a sharded table

Since gsutil does not seem to support bucket rename, what are the alternative ways to accomplish this?

Answer 1

Object names are actually flat in Google Cloud Storage; from the service's perspective, '/' is just another character in the name. The folder abstraction is provided by clients, like gsutil and various GUI tools. Renaming a folder requires clients to request a sequence of copy and delete operations on each object in the folder. There is no atomic way to rename a folder.

Mike Schwartz, Google Cloud Storage team