Fixing MongoDB $lookup aggregation

“Why can’t I match anything when I use $lookup?” is a regular question we get to hear. Almost as much as “Why does everything match when I use $lookup?”. Both scenarios are down to innate MongoDB behavior – and once we show you how you’ll never experience these MongoDB $lookup aggregation issues again. Let’s see how you can use Studio 3T to solve this.

Getting everything with $lookup

Let’s look at an example. Let’s have two tables: makers, and projects. Makers is just a collection of names of people who make things:

{ 
    "_id" : ObjectId("627b6579c088c42c5c44edf9"), 
    "name" : "Bob"
}
{ 
    "_id" : ObjectId("627b68e0c088c42c5c44ee2b"), 
    "name" : "Fred"
}

And projects is simply a collection of projects and the id of the maker who makes it:

{ 
    "_id" : ObjectId("627b65d1c088c42c5c44ee00"), 
    "name" : "Portable digital radio", 
    "maker" : "627b6579c088c42c5c44edf9"
}
{ 
    "_id" : ObjectId("627b6603c088c42c5c44ee05"), 
    "name" : "LED Light Show", 
    "maker" : "627b68e0c088c42c5c44ee2b"
}

Now, if we wanted an aggregation of all the makers and their projects, we’d use the MongoDB $lookup aggregation stage. We’d run the aggregation on the makers collection and add a lookup stage to that which looked up the id in the projects collection, like so:

{
    from: "projects",
    localField: "id",
    foreignField: "made",
    as: "made"
}

That should do but when we run it:

{ 
    "_id" : ObjectId("627b6579c088c42c5c44edf9"), 
    "name" : "Bob", 
    "made" : [
        {
            "_id" : ObjectId("627b65d1c088c42c5c44ee00"), 
            "name" : "Portable digital radio", 
            "maker" : "627b6579c088c42c5c44edf9"
        }, 
        {
            "_id" : ObjectId("627b6603c088c42c5c44ee05"), 
            "name" : "LED Light Show", 
            "maker" : "627b68e0c088c42c5c44ee2b"
        }
    ]
}
{ 
    "_id" : ObjectId("627b68e0c088c42c5c44ee2b"), 
    "name" : "Fred", 
    "made" : [
        {
            "_id" : ObjectId("627b65d1c088c42c5c44ee00"), 
            "name" : "Portable digital radio", 
            "maker" : "627b6579c088c42c5c44edf9"
        }, 
        {
            "_id" : ObjectId("627b6603c088c42c5c44ee05"), 
            "name" : "LED Light Show", 
            "maker" : "627b68e0c088c42c5c44ee2b"
        }
    ]
}

Oh no, every maker has everyone’s projects listed against them.

Why everything?

This is the result of an error we made which doesn’t leap out at you. When we put our $lookup stage together we created it with bad fields:

{
    from: "projects",
    localField: "id",
    foreignField: "made",
    as: "made"
}

First, there is no localField called id. It’s called _id not id. When you use a field that doesn’t exist, MongoDB returns null as the value to allow for documents with varying schemas. 

We made a similar error with the foreignField. There we are looking for made, not maker which is the actual field name. That also returns null.

We now are looking up null and everything in the foreign collection has a null value. So everything matches. And that’s why everything appears in the resulting looked-up arrays. 

Using Studio 3T, don’t forget you can document your schema using the Schema Explorer, so you can see when a field only appears occasionally (or not at all as in this case). 

So, let’s simply use the correct field names:

{
    from: "projects",
    localField: "_id",
    foreignField: "maker",
    as: "made"
}

We press run on the aggregations and we get:

{ 
    "_id" : ObjectId("627b6579c088c42c5c44edf9"), 
    "name" : "Bob", 
    "made" : [

    ]
}
{ 
    "_id" : ObjectId("627b68e0c088c42c5c44ee2b"), 
    "name" : "Fred", 
    "made" : [

    ]
}

Oh no….

Where’s everything gone? 

We now have the exact opposite problem, nothing is appearing in our looked-up arrays. If we look at the collections, we’re sure that the maker fields match the _id fields in the makers collection. So what’s going wrong?

It’s all about types. When you look at the _id field it is usually rendered visually as a string, but to make it efficient, there’s an ObjectId data type, specifically for this kind of value. It’s used by default when you let MongoDB create new records without specifying an _id field.

But it isn’t a string. So when you create another record referring to that document, it needs to be another ObjectID. If your reference is a string, although they may visually look similar, an ObjectID will never be equal to the string.

Fixing this problem with Studio 3T is easy. Just view the makers collection, select the makers field. Then bring up the right click menu and select Field > Edit Value / Type. On the dialog that pops up, set the Type to ObjectId. 

Then select “All documents in collection” on Set this field in. We’re safe selecting that for this collection because we know all documents have a maker field. 

When we’ve selected that, then we can select Only change type (“Convert”) which will use the existing value of each field and convert it to ObjectId. 

Now, click Set Value and Studio 3T will convert all the strings to ObjectIds. You can learn more about changing field types (and how to find erroneous types) in the Knowledge Base article The Quickest Way to Change a Field Type in MongoDB.

Now, if we re-run our aggregation, we will see that the reference is working correctly:

{ 
    "_id" : ObjectId("627b6579c088c42c5c44edf9"), 
    "name" : "Bob", 
    "made" : [
        {
            "_id" : ObjectId("627b65d1c088c42c5c44ee00"), 
            "name" : "Portable digital radio", 
            "maker" : ObjectId("627b6579c088c42c5c44edf9")
        }
    ]
}
{ 
    "_id" : ObjectId("627b68e0c088c42c5c44ee2b"), 
    "name" : "Fred", 
    "made" : [
        {
            "_id" : ObjectId("627b6603c088c42c5c44ee05"), 
            "name" : "LED Light Show", 
            "maker" : ObjectId("627b68e0c088c42c5c44ee2b")
        }
    ]
}

Both Bob and Fred have their own projects, and we can see the maker field is indeed, now an ObjectId.

MongoDB $lookup aggregation – Lessons learned

So, we’ve shown you how getting field names wrong can lead to silent errors and too much data being returned, how you can use Schema Explorer to find correct field names, why trying to match fields with different types can go wrong silently too and how you can use Studio 3T to fix your data types quickly.