How to keep a Firebase database sync with BigQuery?

Question

We are working on a project where a lot of data is involved. Now we recently read about Google BigQuery. But how can we export the data to this platform? We have seen the sample of importing logs into Google BigQuery. But this does not contain information about updating and deleting data (only inserting).

So our objects are able to update their data. And we have a limited amount of queries on the BigQuery tables. How can we synchronize our data without exceeding the BigQuery quota limits.

Our current function code:

'use strict';

// Default imports.

const functions = require('firebase-functions');
const bigQuery = require('@google-cloud/bigquery')();

// If you want to change the nodes to listen to REMEMBER TO change the constants below.
// The 'id' field is AUTOMATICALLY added to the values, so you CANNOT add it.

const ROOT_NODE = 'categories';
const VALUES = [
    'name'
];

// This function listens to the supplied root node.
// When the root node is completed empty all of the Google BigQuery rows will be removed.
// This function should only activate when the root node is deleted.

exports.root = functions.database.ref(ROOT_NODE).onWrite(event => {
    if (event.data.exists()) {
        return;
    }

    return bigQuery.query({
        query: [
            'DELETE FROM `stampwallet.' + ROOT_NODE + '`',
            'WHERE true'
        ].join(' '),
        params: []
    });
});

// This function listens to the supplied root node, but on child added/removed/changed.
// When an object is inserted/deleted/updated the appropriate action will be taken.

exports.children = functions.database.ref(ROOT_NODE + '/{id}').onWrite(event => {
    const id = event.params.id;

    if (!event.data.exists()) {
        return bigQuery.query({
            query: [
                'DELETE FROM `stampwallet.' + ROOT_NODE + '`',
                'WHERE id = ?'
            ].join(' '),
            params: [
                id
            ]
        });
    }

    const item = event.data.val();

    if (event.data.previous.exists()) {
        let update = [];
        for (let index = 0; index < VALUES.length; index++) {
            const value = VALUES[index];

            update.push(item[value]);
        }
        update.push(id);

        return bigQuery.query({
            query: [
                'UPDATE `stampwallet.' + ROOT_NODE + '`',
                'SET ' + VALUES.join(' = ?, ') + ' = ?',
                'WHERE id = ?'
            ].join(' '),
            params: update
        });
    }

    let template = [];
    for (let index = 0; index < VALUES.length; index++) {
        template.push('?');
    }

    let create = [];
    create.push(id);
    for (let index = 0; index < VALUES.length; index++) {
        const value = VALUES[index];

        create.push(item[value]);
    }

    return bigQuery.query({
        query: [
            'INSERT INTO `stampwallet.' + ROOT_NODE + '` (id, ' + VALUES.join(', ') + ')',
            'VALUES (?, ' + template.join(', ') + ')'
        ].join(' '),
        params: create
    });
});

What would be the best way to sync firebase to bigquery?

Answer 1

BigQuery supports UPDATE and DELETE, but not frequent ones - BigQuery is an analytical database, not a transactional one.

To synchronize a transactional database with BigQuery you can use approaches like:

• Export a daily dump, and import it into BigQuery.
• Treat updates and deletes as new events, and keep appending events to your BigQuery event log.
• Use a tool like https://github.com/MemedDev/mysql-to-google-bigquery.
• Approaches like "BigQuery at WePay part III: Automating MySQL exports every 15 minutes with Airflow, and dealing with updates"

With Firebase you could schedule a daily load to BigQuery from their daily backups:

• https://firebase.googleblog.com/2016/10/announcing-automated-daily-backups-for-the-firebase-database.html