Skip to content
Studio 3T - The professional GUI, IDE and client for MongoDB
  • Tools
    • Aggregation Editor
    • IntelliShell
    • Visual Query Builder
    • Export Wizard
    • Import Wizard
    • Query Code
    • SQL Query
    • Connect
    • Schema Explorer
    • Compare
    • SQL ⇔ MongoDB Migration
    • Data Masking
    • Task Scheduler
    • Reschema
    • More Tools and Features
  • Solutions
  • Resources
    • Knowledge Base
    • MongoDB Tutorials & Courses
    • Tool/Feature Documentation
    • Reports
    • Case Studies
    • Whitepapers
    • Blog
    • Testimonials
    • Community
  • Contact us
    • Contact
    • Sales Support
    • Feedback and Support
    • Career
    • About Us
  • Store
    • Buy Now
    • Preferred Resellers
    • Team Pricing
  • My License
  • Download
search

Fixing MongoDB $lookup aggregation

Posted on: 25/05/2022 by DJ Walker-Morgan

“Why can’t I match anything when I use $lookup?” is a regular question we get to hear. Almost as much as “Why does everything match when I use $lookup?”. Both scenarios are down to innate MongoDB behavior – and once we show you how you’ll never experience these MongoDB $lookup aggregation issues again. Let’s see how you can use Studio 3T to solve this.

Getting everything with $lookup

Let’s look at an example. Let’s have two tables: makers, and projects. Makers is just a collection of names of people who make things:

{ 
    "_id" : ObjectId("627b6579c088c42c5c44edf9"), 
    "name" : "Bob"
}
{ 
    "_id" : ObjectId("627b68e0c088c42c5c44ee2b"), 
    "name" : "Fred"
}

And projects is simply a collection of projects and the id of the maker who makes it:

{ 
    "_id" : ObjectId("627b65d1c088c42c5c44ee00"), 
    "name" : "Portable digital radio", 
    "maker" : "627b6579c088c42c5c44edf9"
}
{ 
    "_id" : ObjectId("627b6603c088c42c5c44ee05"), 
    "name" : "LED Light Show", 
    "maker" : "627b68e0c088c42c5c44ee2b"
}

Now, if we wanted an aggregation of all the makers and their projects, we’d use the MongoDB $lookup aggregation stage. We’d run the aggregation on the makers collection and add a lookup stage to that which looked up the id in the projects collection, like so:

{
    from: "projects",
    localField: "id",
    foreignField: "made",
    as: "made"
}

That should do but when we run it:

{ 
    "_id" : ObjectId("627b6579c088c42c5c44edf9"), 
    "name" : "Bob", 
    "made" : [
        {
            "_id" : ObjectId("627b65d1c088c42c5c44ee00"), 
            "name" : "Portable digital radio", 
            "maker" : "627b6579c088c42c5c44edf9"
        }, 
        {
            "_id" : ObjectId("627b6603c088c42c5c44ee05"), 
            "name" : "LED Light Show", 
            "maker" : "627b68e0c088c42c5c44ee2b"
        }
    ]
}
{ 
    "_id" : ObjectId("627b68e0c088c42c5c44ee2b"), 
    "name" : "Fred", 
    "made" : [
        {
            "_id" : ObjectId("627b65d1c088c42c5c44ee00"), 
            "name" : "Portable digital radio", 
            "maker" : "627b6579c088c42c5c44edf9"
        }, 
        {
            "_id" : ObjectId("627b6603c088c42c5c44ee05"), 
            "name" : "LED Light Show", 
            "maker" : "627b68e0c088c42c5c44ee2b"
        }
    ]
}

Oh no, every maker has everyone’s projects listed against them.

Why everything?

This is the result of an error we made which doesn’t leap out at you. When we put our $lookup stage together we created it with bad fields:

{
    from: "projects",
    localField: "id",
    foreignField: "made",
    as: "made"
}

First, there is no localField called id. It’s called _id not id. When you use a field that doesn’t exist, MongoDB returns null as the value to allow for documents with varying schemas. 

We made a similar error with the foreignField. There we are looking for made, not maker which is the actual field name. That also returns null.

We now are looking up null and everything in the foreign collection has a null value. So everything matches. And that’s why everything appears in the resulting looked-up arrays. 

Using Studio 3T, don’t forget you can document your schema using the Schema Explorer, so you can see when a field only appears occasionally (or not at all as in this case). 

So, let’s simply use the correct field names:

{
    from: "projects",
    localField: "_id",
    foreignField: "maker",
    as: "made"
}

We press run on the aggregations and we get:

{ 
    "_id" : ObjectId("627b6579c088c42c5c44edf9"), 
    "name" : "Bob", 
    "made" : [

    ]
}
{ 
    "_id" : ObjectId("627b68e0c088c42c5c44ee2b"), 
    "name" : "Fred", 
    "made" : [

    ]
}

Oh no….

Where’s everything gone? 

We now have the exact opposite problem, nothing is appearing in our looked-up arrays. If we look at the collections, we’re sure that the maker fields match the _id fields in the makers collection. So what’s going wrong?

It’s all about types. When you look at the _id field it is usually rendered visually as a string, but to make it efficient, there’s an ObjectId data type, specifically for this kind of value. It’s used by default when you let MongoDB create new records without specifying an _id field.

But it isn’t a string. So when you create another record referring to that document, it needs to be another ObjectID. If your reference is a string, although they may visually look similar, an ObjectID will never be equal to the string.

Fixing this problem with Studio 3T is easy. Just view the makers collection, select the makers field. Then bring up the right click menu and select Field > Edit Value / Type. On the dialog that pops up, set the Type to ObjectId. 

Then select “All documents in collection” on Set this field in. We’re safe selecting that for this collection because we know all documents have a maker field. 

When we’ve selected that, then we can select Only change type (“Convert”) which will use the existing value of each field and convert it to ObjectId. 

Screenshot of the Edit Value dialog for changing a field type - we do this so the types match and MongoDB can do an efficient and correct $lookup aggregation stage.
Changing a field’s type with Studio 3T

Now, click Set Value and Studio 3T will convert all the strings to ObjectIds. You can learn more about changing field types (and how to find erroneous types) in the Knowledge Base article The Quickest Way to Change a Field Type in MongoDB.

Now, if we re-run our aggregation, we will see that the reference is working correctly:

{ 
    "_id" : ObjectId("627b6579c088c42c5c44edf9"), 
    "name" : "Bob", 
    "made" : [
        {
            "_id" : ObjectId("627b65d1c088c42c5c44ee00"), 
            "name" : "Portable digital radio", 
            "maker" : ObjectId("627b6579c088c42c5c44edf9")
        }
    ]
}
{ 
    "_id" : ObjectId("627b68e0c088c42c5c44ee2b"), 
    "name" : "Fred", 
    "made" : [
        {
            "_id" : ObjectId("627b6603c088c42c5c44ee05"), 
            "name" : "LED Light Show", 
            "maker" : ObjectId("627b68e0c088c42c5c44ee2b")
        }
    ]
}

Both Bob and Fred have their own projects, and we can see the maker field is indeed, now an ObjectId.

MongoDB $lookup aggregation – Lessons learned

So, we’ve shown you how getting field names wrong can lead to silent errors and too much data being returned, how you can use Schema Explorer to find correct field names, why trying to match fields with different types can go wrong silently too and how you can use Studio 3T to fix your data types quickly. 


How helpful was this article?
This article was hideous
This article was bad
This article was ok
This article was good
This article was great
Thank you for your feedback!

About The Author

DJ Walker-Morgan

Dj has been around since Commodore had Pets and Apples grew everywhere. With a background in Unix and development, he's been around the technology business writing code or writing content ever since.

Related articles

  • MongoDB $lookup Example | The MongoDB Aggregation Pipeline
  • Adding Lookup Data to the Aggregation Pipeline: Test your skills
  • Adding Lookup Data to the Aggregation Pipeline
  • Exercise 1: Adding lookup data to the aggregation pipeline
  • Exercise 2: Converting string values in one of the lookup fields to integers

Tags

academy aggregation AMA atlas community connections date tags events export features hackolade import intellishell JSON knowledge base migration modelling mongodb mongoodb productivity query regex releases schema security SQL Studio 3T tasks tips updates webinar windows

Browse by MongoDB topic

  • Connecting to MongoDB
  • Database Administration & Security
  • Getting Started with MongoDB
  • Getting Started with Studio 3T
  • Import/Export
  • Job Automation & Scheduling
  • MongoDB Aggregation Framework
  • MongoDB/Studio 3T Workshops
  • Performance
  • Query & CRUD Operations
  • Reference
  • Schema
  • Studio 3T Licensing
  • Support and other resources
  • Working with MongoDB & SQL
  • Working with MongoDB Atlas

Studio 3T

MongoDB Enterprise Certified Technology PartnerSince 2014, 3T has been helping thousands of MongoDB developers and administrators with their everyday jobs by providing the finest MongoDB tools on the market. We guarantee the best compatibility with current and legacy releases of MongoDB, continue to deliver new features with every new software release, and provide high quality support.

Find us on FacebookFind us on TwitterFind us on YouTubeFind us on LinkedIn

Education

  • Free MongoDB Tutorials
  • Connect to MongoDB
  • Connect to MongoDB Atlas
  • Import Data to MongoDB
  • Export MongoDB Data
  • Build Aggregation Queries
  • Query MongoDB with SQL
  • Migrate from SQL to MongoDB

Resources

  • Feedback and Support
  • Sales Support
  • Knowledge Base
  • FAQ
  • Reports
  • Case Studies
  • White Papers
  • Testimonials
  • Discounts

Company

  • About Us
  • Blog
  • Careers
  • Legal
  • Press
  • Privacy Policy
  • EULA

© 2022 3T Software Labs GmbH. All rights reserved.

  • Privacy Policy
  • Cookie settings
  • Impressum
When you click "Accept", you are agreeing to cookies being on your device. They may improve site navigation, site usage analysis, or the relevance of messages. It is up to you which cookies are enabled. Read our Privacy Policy.
Manage cookies
Accept
✕

Privacy Preference Center

A cookie is a small file of letters and numbers that is downloaded on to your computer when you visit a website. Cookies are used by many websites and can do a number of things, eg remembering your preferences, recording what you have put in your shopping basket, and counting the number of people looking at a website. In the form below you can choose which cookies, except for essential cookies, to allow or disable.

Accept all cookies

Manage consent preferences

Essential cookies are strictly necessary to provide an online service such as our website or a service on our website which you have requested. The website or service will not work without them.

Performance cookies allow us to collect information such as number of visits and sources of traffic. This information is used in aggregate form to help us understand how our websites are being used, allowing us to improve both our website’s performance and your experience.

Functional cookies collect information about your preferences and choices and make using the website a lot easier and more relevant. Without these cookies, some of the site functionality may not work as intended.

Social media cookies are cookies used to share user behaviour information with a third-party social media platform. They may consequently effect how social media sites present you with information in the future.

Accept selected