Skip to content
Studio 3T - The professional GUI, IDE and client for MongoDB
  • Tools
    • Aggregation Editor
    • IntelliShell
    • Visual Query Builder
    • Export Wizard
    • Import Wizard
    • Query Code
    • SQL Query
    • Connect
    • Schema Explorer
    • Compare
    • SQL ⇔ MongoDB Migration
    • Data Masking
    • Task Scheduler
    • Reschema
    • More Tools and Features
  • Solutions
  • Resources
    • Knowledge Base
    • MongoDB Tutorials & Courses
    • Tool/Feature Documentation
    • Blog
    • Community
    • Testimonials
    • Whitepapers
    • Reports
  • Contact us
    • Contact
    • Sales Support
    • Feedback and Support
    • Careers
    • About Us
  • Store
    • Buy Now
    • Preferred Resellers
    • Team Pricing
  • Download
  • My 3T
search

Studio 3T® Knowledge Base

  • Documentation
  • Tutorials
  • Workshops
Take the fastest route to learning MongoDB. Cover the basics in two hours with MongoDB 101, no registration required.
Start the free course

Incremental Execution for Export and Migration

Posted on: 06/01/2022 (last updated: 18/01/2022) by Dj Walker-Morgan

Incremental execution is a feature of Studio 3T which exists in Export and MongoDB to SQL Migrations. Incremental execution’s design is to make these operations much more manageable.

Let’s start with the big picture: State. Incremental execution records the state of a particular field as it passes through the Export or Migration process. That field has to be both unique and sortable. As each document passes through, the state is updated and continues until the Export or Migration has run out of records to process. The next time that the task is run, Studio 3T picks up on that recorded state and transparently uses it to reshape the Export/Migration’s underlying query and allow it to restart the process from the document after the one recorded. 

Practical Incremental Execution

If that’s a little abstract, let’s look at an application for Incremental Execution: Batched exporting. This is where we want export data regularly in batches of, say, a thousand. To do this, we start with the export query; we only want to present a thousand documents to the Export, so we’ll create a query with a limit of a thousand.

In this example, we are exporting to a file in this example and using Studio 3T placeholders to generate a new file each time we export so we can see the results. 

Below the query, you’ll see the Incremental execution controls:

Incremental execution is disabled because we have to configure it. Clicking on Configure incremental execution will bring up the configuration dialog. Here, we want to enable incremental execution and set the Resume Point options to start with a fresh state. We’ll use the default, _id, as the field to track.

Why _id? As a generated ObjectID it is pretty much guaranteed to be unique and sortable. It is also has an index by default. Click OK and we’ll be back at the export unit configuration. Now, something has changed:

This tells us we have configured incremental execution. But because this Export unit has not been run, there is no state for it to refer to.

Running An Export With Incremental Execution

If we Run this unit, it will export the 1000 documents we specified in the query’s limit, and if we look over to the output directory, there will be one file, with 1000 documents in it.

If we look back at the Export unit’s configuration, we’ll notice another change:

This shows us the Export unit has retained state from the export run. When you run the Export unit next, it will use the next _id that follows. If we click Run again, a new export file of 1000 documents will appear and the state will update. As there’s only 5000 documents in the collection, another three runs will exhaust the collection. With no more data available, the next Export run will write a blank file as there are no new records. If you copy and paste some records into the collection, then run the export again, it’ll pick up on these new records and export just those.

Underneath The Options

What’s happening is a transparent modification of the query. Incremental execution starts an export or migration by sorting the collection by the specified field. This sorting allows incremental execution to locate the “first” document and then work through the collection in a known order. 

The first time the unit is run it will start at the “first” document of the sorted collection. The next time it is run, there is now a state, and it’s here that incremental execution transparently adds in an and with a { $gt: value } clause to the query, where the value is the saved state. This allows the query to resume, skipping the previously processed documents and then start on the next batch. 

Resetting The State

There are times when it may be necessary to reset the recorded state, back in time to a previous run. The incremental execution options have you covered. If you bring the options up after a few runs of the export, you’ll notice that the Select a resume point option is available. The drop down menu will list the last five runs, when they happened and what state they were in. Select one and on the next run, the Export or Migration will start from that state. Or, if you want to start all over again, select Start from a fresh state. This will clear the state and all the documents returned by the export unit’s query will be eligible for export.

Incremental Execution With Databases

While the example above uses files, to make it easy to see the results, Export and Migration tasks can also target databases and collections. Incremental execution works in exactly the same way with these targets. It does not take into account updated records in the source collection. Incremental Execution’s design focuses on tracking the new records in the database since the last time a task was run. Although a database may have updates to existing documents, incremental execution will not act as a source of updates. 

Using Other Fields

In the examples, we’ve used the _id field because it is sortable, unique and used in every MongoDB collection. Incremental execution can use other fields, as long as they are sortable and unique, and ideally, immutable. If the field is mutable, that is can be changed, then the order of documents can change over time and documents previously exported could turn up again as eligible for export.


How helpful was this article?
This article was hideous
This article was bad
This article was ok
This article was good
This article was great
Thank you for your feedback!

About The Author

Dj Walker-Morgan

Dj has been around since Commodore had Pets and Apples grew everywhere. With a background in Unix and development, he's been around the technology business writing code or writing content ever since.

Article navigation

Related articles

  • What’s New in Studio 3T 2021.7 | Incremental migrations, customizable SQL types, and more
  • What’s New in Studio 3T 2020.4 | Introducing License Manager, IBM DB2 Migration Support, Export Wizard Improvements
  • Studio 3T SQL Export or Migration? #STUDIO3T_AMA
  • What’s New in Studio 3T 2020.7 | Export Documents from Table View to CSV, Edit find() Queries During Export & More
  • SQL Migration: MongoDB and SQL, Coexistence Made Easy

Studio 3T

MongoDB Enterprise Certified Technology PartnerSince 2014, 3T has been helping thousands of MongoDB developers and administrators with their everyday jobs by providing the finest MongoDB tools on the market. We guarantee the best compatibility with current and legacy releases of MongoDB, continue to deliver new features with every new software release, and provide high quality support.

Find us on FacebookFind us on TwitterFind us on YouTubeFind us on LinkedIn

Education

  • Free MongoDB Tutorials
  • Connect to MongoDB
  • Connect to MongoDB Atlas
  • Import Data to MongoDB
  • Export MongoDB Data
  • Build Aggregation Queries
  • Query MongoDB with SQL
  • Migrate from SQL to MongoDB

Resources

  • Feedback and Support
  • Sales Support
  • Knowledge Base
  • FAQ
  • Reports
  • White Papers
  • Testimonials
  • Discounts

Company

  • About Us
  • Blog
  • Careers
  • Legal
  • Press
  • Privacy Policy
  • EULA

© 2023 3T Software Labs Ltd. All rights reserved.

  • Privacy Policy
  • Cookie settings
  • Impressum

We value your privacy

With your consent, we and third-party providers use cookies and similar technologies on our website to analyse your use of our site for market research or advertising purposes ("analytics and marketing") and to provide you with additional functions (“functional”). This may result in the creation of pseudonymous usage profiles and the transfer of personal data to third countries, including the USA, which may have no adequate level of protection for the processing of personal data.

By clicking “Accept all”, you consent to the storage of cookies and the processing of personal data for these purposes, including any transfers to third countries. By clicking on “Decline all”, you do not give your consent and we will only store cookies that are necessary for our website. You can customize the cookies we store on your device or change your selection at any time - thus also revoking your consent with effect for the future - under “Manage Cookies”, or “Cookie Settings” at the bottom of the page. You can find further information in our Privacy Policy.
Accept all
Decline all
Manage cookies
✕

Privacy Preference Center

With your consent, we and third-party providers use cookies and similar technologies on our website to analyse your use of our site for market research or advertising purposes ("analytics and marketing") and to provide you with additional functions (“functional”). This may result in the creation of pseudonymous usage profiles and the transfer of personal data to third countries, including the USA, which may have no adequate level of protection for the processing of personal data. Please choose for which purposes you wish to give us your consent and store your preferences by clicking on “Accept selected”. You can find further information in our Privacy Policy.

Accept all cookies

Manage consent preferences

Essential cookies are strictly necessary to provide an online service such as our website or a service on our website which you have requested. The website or service will not work without them.

Performance cookies allow us to collect information such as number of visits and sources of traffic. This information is used in aggregate form to help us understand how our websites are being used, allowing us to improve both our website’s performance and your experience.

Google Analytics

Google Ads

Bing Ads

Facebook

LinkedIn

Quora

Hotjar

Reddit

Functional cookies collect information about your preferences and choices and make using the website a lot easier and more relevant. Without these cookies, some of the site functionality may not work as intended.

HubSpot

Social media cookies are cookies used to share user behaviour information with a third-party social media platform. They may consequently effect how social media sites present you with information in the future.

Accept selected