MongoDB: Run Aggregate on multiple databases

Question

I have many number of databases each containing data for each customer.

Is there any way to run an Aggregate which will hit multiple databases? I want to run $group on the complete dataset.

As a workaround, I'm putting the data from both of the DBs into a temporary collection and then running the $group on top of this temp collection.

Is there any better way to this? I'm open to use any other technology as well!

Answer 1

There is no way to have the aggregation pipeline operate on multiple databases simultaneously. MongoDB aggregation may perform a $lookup between collections to perform a join, but this is the extent to which it's possible to merge external data. Unfortunately manually merging data--either merging raw data into a single collection to perform a single aggregation or merging the results of multiple aggregations--is the only option available to you if you continue to use your existing architecture.

If you modify your database architecture to contain all customer data within the same database and restrict access based on e.g. a customerId field, then it would be possible to perform aggregation on all customer data simultaneously without any additional effort. As it stands, however, your existing architecture simply will not work for this.