MongoDB: Combine data from multiple collections into one..how?

MongoDB 3.2 now allows one to combine data from multiple collections into one through the $lookup aggregation stage. As a practical example, lets say that you have data about books split into two different collections.

First collection, called books, having the following data:

{    "isbn": "978-3-16-148410-0",    "title": "Some cool book",    "author": "John Doe"}{    "isbn": "978-3-16-148999-9",    "title": "Another awesome book",    "author": "Jane Roe"}

And the second collection, called books_selling_data, having the following data:

{    "_id": ObjectId("56e31bcf76cdf52e541d9d26"),    "isbn": "978-3-16-148410-0",    "copies_sold": 12500}{    "_id": ObjectId("56e31ce076cdf52e541d9d28"),    "isbn": "978-3-16-148999-9",    "copies_sold": 720050}{    "_id": ObjectId("56e31ce076cdf52e541d9d29"),    "isbn": "978-3-16-148999-9",    "copies_sold": 1000}

To merge both collections is just a matter of using $lookup in the following way:

db.books.aggregate([{    $lookup: {            from: "books_selling_data",            localField: "isbn",            foreignField: "isbn",            as: "copies_sold"        }}])

After this aggregation, the books collection will look like the following:

{    "isbn": "978-3-16-148410-0",    "title": "Some cool book",    "author": "John Doe",    "copies_sold": [        {            "_id": ObjectId("56e31bcf76cdf52e541d9d26"),            "isbn": "978-3-16-148410-0",            "copies_sold": 12500        }    ]}{    "isbn": "978-3-16-148999-9",    "title": "Another awesome book",    "author": "Jane Roe",    "copies_sold": [        {            "_id": ObjectId("56e31ce076cdf52e541d9d28"),            "isbn": "978-3-16-148999-9",            "copies_sold": 720050        },        {            "_id": ObjectId("56e31ce076cdf52e541d9d28"),            "isbn": "978-3-16-148999-9",            "copies_sold": 1000        }    ]}

It is important to note a few things:

1. The "from" collection, in this case books_selling_data, cannot be sharded.
2. The "as" field will be an array, as the example above.
3. Both "localField" and "foreignField" options on the $lookup stage will be treated as null for matching purposes if they don't exist in their respective collections (the $lookup docs has a perfect example about that).

So, as a conclusion, if you want to consolidate both collections, having, in this case, a flat copies_sold field with the total copies sold, you will have to work a little bit more, probably using an intermediary collection that will, then, be $out to the final collection.

Although you can't do this real-time, you can run map-reduce multiple times to merge data together by using the "reduce" out option in MongoDB 1.8+ map/reduce (see http://www.mongodb.org/display/DOCS/MapReduce#MapReduce-Outputoptions). You need to have some key in both collections that you can use as an _id.

For example, let's say you have a users collection and a comments collection and you want to have a new collection that has some user demographic info for each comment.

Let's say the users collection has the following fields:

• _id
• firstName
• lastName
• country
• gender
• age

And then the comments collection has the following fields:

• _id
• userId
• comment
• created

You would do this map/reduce:

var mapUsers, mapComments, reduce;db.users_comments.remove();// setup sample data - wouldn't actually use this in productiondb.users.remove();db.comments.remove();db.users.save({firstName:"Rich",lastName:"S",gender:"M",country:"CA",age:"18"});db.users.save({firstName:"Rob",lastName:"M",gender:"M",country:"US",age:"25"});db.users.save({firstName:"Sarah",lastName:"T",gender:"F",country:"US",age:"13"});var users = db.users.find();db.comments.save({userId: users[0]._id, "comment": "Hey, what's up?", created: new ISODate()});db.comments.save({userId: users[1]._id, "comment": "Not much", created: new ISODate()});db.comments.save({userId: users[0]._id, "comment": "Cool", created: new ISODate()});// end sample data setupmapUsers = function() {    var values = {        country: this.country,        gender: this.gender,        age: this.age    };    emit(this._id, values);};mapComments = function() {    var values = {        commentId: this._id,        comment: this.comment,        created: this.created    };    emit(this.userId, values);};reduce = function(k, values) {    var result = {}, commentFields = {        "commentId": '',         "comment": '',        "created": ''    };    values.forEach(function(value) {        var field;        if ("comment" in value) {            if (!("comments" in result)) {                result.comments = [];            }            result.comments.push(value);        } else if ("comments" in value) {            if (!("comments" in result)) {                result.comments = [];            }            result.comments.push.apply(result.comments, value.comments);        }        for (field in value) {            if (value.hasOwnProperty(field) && !(field in commentFields)) {                result[field] = value[field];            }        }    });    return result;};db.users.mapReduce(mapUsers, reduce, {"out": {"reduce": "users_comments"}});db.comments.mapReduce(mapComments, reduce, {"out": {"reduce": "users_comments"}});db.users_comments.find().pretty(); // see the resulting collection

At this point, you will have a new collection called users_comments that contains the merged data and you can now use that. These reduced collections all have _id which is the key you were emitting in your map functions and then all of the values are a sub-object inside the value key - the values aren't at the top level of these reduced documents.

This is a somewhat simple example. You can repeat this with more collections as much as you want to keep building up the reduced collection. You could also do summaries and aggregations of data in the process. Likely you would define more than one reduce function as the logic for aggregating and preserving existing fields gets more complex.

You'll also note that there is now one document for each user with all of that user's comments in an array. If we were merging data that has a one-to-one relationship rather than one-to-many, it would be flat and you could simply use a reduce function like this:

reduce = function(k, values) {    var result = {};    values.forEach(function(value) {        var field;        for (field in value) {            if (value.hasOwnProperty(field)) {                result[field] = value[field];            }        }    });    return result;};

If you want to flatten the users_comments collection so it's one document per comment, additionally run this:

var map, reduce;map = function() {    var debug = function(value) {        var field;        for (field in value) {            print(field + ": " + value[field]);        }    };    debug(this);    var that = this;    if ("comments" in this.value) {        this.value.comments.forEach(function(value) {            emit(value.commentId, {                userId: that._id,                country: that.value.country,                age: that.value.age,                comment: value.comment,                created: value.created,            });        });    }};reduce = function(k, values) {    var result = {};    values.forEach(function(value) {        var field;        for (field in value) {            if (value.hasOwnProperty(field)) {                result[field] = value[field];            }        }    });    return result;};db.users_comments.mapReduce(map, reduce, {"out": "comments_with_demographics"});

This technique should definitely not be performed on the fly. It's suited for a cron job or something like that which updates the merged data periodically. You'll probably want to run ensureIndex on the new collection to make sure queries you perform against it run quickly (keep in mind that your data is still inside a value key, so if you were to index comments_with_demographics on the comment created time, it would be db.comments_with_demographics.ensureIndex({"value.created": 1});

Doing unions in MongoDB in a 'SQL UNION' fashion is possible using aggregations along with lookups, in a single query. Here is an example I have tested that works with MongoDB 4.0:

// Create employees data for testing the union.db.getCollection('employees').insert({ name: "John", type: "employee", department: "sales" });db.getCollection('employees').insert({ name: "Martha", type: "employee", department: "accounting" });db.getCollection('employees').insert({ name: "Amy", type: "employee", department: "warehouse" });db.getCollection('employees').insert({ name: "Mike", type: "employee", department: "warehouse"  });// Create freelancers data for testing the union.db.getCollection('freelancers').insert({ name: "Stephany", type: "freelancer", department: "accounting" });db.getCollection('freelancers').insert({ name: "Martin", type: "freelancer", department: "sales" });db.getCollection('freelancers').insert({ name: "Doug", type: "freelancer", department: "warehouse"  });db.getCollection('freelancers').insert({ name: "Brenda", type: "freelancer", department: "sales"  });// Here we do a union of the employees and freelancers using a single aggregation query.db.getCollection('freelancers').aggregate( // 1. Use any collection containing at least one document.  [    { $limit: 1 }, // 2. Keep only one document of the collection.    { $project: { _id: '$$REMOVE' } }, // 3. Remove everything from the document.    // 4. Lookup collections to union together.    { $lookup: { from: 'employees', pipeline: [{ $match: { department: 'sales' } }], as: 'employees' } },    { $lookup: { from: 'freelancers', pipeline: [{ $match: { department: 'sales' } }], as: 'freelancers' } },    // 5. Union the collections together with a projection.    { $project: { union: { $concatArrays: ["$employees", "$freelancers"] } } },    // 6. Unwind and replace root so you end up with a result set.    { $unwind: '$union' },    { $replaceRoot: { newRoot: '$union' } }  ]);

Here is the explanation of how it works:

1. Instantiate an aggregate out of any collection of your database that has at least one document in it. If you can't guarantee any collection of your database will not be empty, you can workaround this issue by creating in your database some sort of 'dummy' collection containing a single empty document in it that will be there specifically for doing union queries.

2. Make the first stage of your pipeline to be { $limit: 1 }. This will strip all the documents of the collection except the first one.

3. Strip all the fields of the remaining document by using a $project stage:

   { $project: { _id: '$$REMOVE' } }

4. Your aggregate now contains a single, empty document. It's time to add lookups for each collection you want to union together. You may use the pipeline field to do some specific filtering, or leave localField and foreignField as null to match the whole collection.

   { $lookup: { from: 'collectionToUnion1', pipeline: [...], as: 'Collection1' } },{ $lookup: { from: 'collectionToUnion2', pipeline: [...], as: 'Collection2' } },{ $lookup: { from: 'collectionToUnion3', pipeline: [...], as: 'Collection3' } }

5. You now have an aggregate containing a single document that contains 3 arrays like this:

   {    Collection1: [...],    Collection2: [...],    Collection3: [...]}

   You can then merge them together into a single array using a $project stage along with the $concatArrays aggregation operator:

   {  "$project" :  {    "Union" : { $concatArrays: ["$Collection1", "$Collection2", "$Collection3"] }  }}

6. You now have an aggregate containing a single document, into which is located an array that contains your union of collections. What remains to be done is to add an $unwind and a $replaceRoot stage to split your array into separate documents:

   { $unwind: "$Union" },{ $replaceRoot: { newRoot: "$Union" } }

7. Voilà. You now have a result set containing the collections you wanted to union together. You can then add more stages to filter it further, sort it, apply skip() and limit(). Pretty much anything you want.