Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Firestore: sync more than one collecttion/subcollection to BigQuery using extension firestore-bigquery-export

I need to sync Firestore and BigQuery for realtime data analysis. My Firestore database has 2 different collections with subcollections and I need to sync all the colections/subcollections. I cannot change this structure.

I test the firestore-bigquery-export extension to sync juste ONE Firestore collection or ONE subcollection (using {wildcard}) and it works very well (except for data location set to US also if select another location).

What's the way to sync more than one collection/subcollection?

I read from extension documentation: https://github.com/firebase/extensions/tree/master/firestore-bigquery-export : note that this extension only listens for document changes in the collection, but no changes in any subcollection. You can, though, install additional instances of this extension to specifically listen to a subcollection or other collections in your database.

How can I install additional instances of extensions?

Otherwise, I could set a cloud function that writes one big collection that joins the 2 collections whenever there is a database edit and syncs the one new collection. But I don't like this solution because it's redundant

like image 211
franco_b Avatar asked Sep 06 '25 02:09

franco_b


1 Answers

I solve. I realize that is possible install more instance of extension directly from Firestore UI. I will'install a many extension as there are the collections/subcollections.

like image 57
franco_b Avatar answered Sep 07 '25 19:09

franco_b