Skip to content
Studio 3T - The professional GUI, IDE and client for MongoDB
  • Tools
    • Aggregation Editor
    • IntelliShell
    • Visual Query Builder
    • Export Wizard
    • Import Wizard
    • Query Code
    • SQL Query
    • Connect
    • Schema Explorer
    • Compare
    • SQL ⇔ MongoDB Migration
    • Data Masking
    • Task Scheduler
    • Reschema
    • AI Helper
    • Team Sharing
    • Index Manager
    • More Tools and Features
  • Solutions
  • Resources
    • Knowledge Base
    • MongoDB Tutorials & Courses
    • Tool/Feature Documentation
    • Blog
    • Community
    • Testimonials
    • Whitepapers
    • Reports
  • Contact us
    • Contact
    • Sales Support
    • Feedback and Support
    • Careers
    • About Us
  • Store
    • Buy Now
    • Preferred Resellers
    • Team Pricing
  • Download
  • My 3T
search

Boosting SQL Performance

Posted on: 18/08/2021 (last updated: 06/09/2021) by Dj Walker-Morgan
Hugo Almeida, Senior Software Developer at 3T Software Labs
Hugo Almeida, Senior Software Developer at 3T Software Labs

One of Studio 3T’s strengths is our support for SQL: for direct queries written in SQL, for simple imports of SQL tables, and also for more complex migrations of entire databases between MongoDB and SQL.

The import and migration tools help data flow smoothly between MongoDB and SQL databases such as MySQL, Microsoft SQL Server, Oracle, Postgresql, Sybase and IBM DB2. In the recent 2021.5 release of Studio 3T, the SQL feature received a performance boost. We wanted to know more, so we sat down with Hugo Almeida, developer lead on the SQL connections.

In one test, importing two million records from Oracle was taking over four hours. In the forthcoming 2021.6 release of Studio 3T, that import took six minutes.

Evaluating Speeds

Was there a problem or was this all standalone performance work? It turns out that if you were migrating a single table, it was fine. But anything with relationships could slow it down. That slowness was in proportion to the distance the Studio 3T system was from the SQL database server, typically expressed in a higher latency in calls.

Hugo explained how the SQL to MongoDB migration tool dealt with building collections.  When there were two tables with a relationship, Studio 3T would get one record from the first table, then lookup the relationship in the second table, build the document, write it and move on to the next record. These simple atomic operations work well with the database, keeping I/O and compute load down. More relationships mean, of course, more secondary queries.

But, while this approach works well with local SQL databases, the addition of  latency into the process had a dramatic effect on the migration performance, pushing what should have been a quick transfer up into the hours or even days. Every lookup ends up waiting for at least the connection latency time to get a response, and the more relationships there are, the more waiting there is.

Doing the math

“If you just do simple math,” Hugo explained, “now, imagine each round trip is like two seconds – which it isn’t, but for simplicity’s sake let’s say that – and the query itself takes one second, then it takes three seconds to get a result. Then for 100 records, that would be three hundred seconds”. Give each record two related tables with relationships, you are now up to 900 seconds. And a vast amount of that time is waiting for the request to travel across the Internet.

Now, one approach might be to make the relationships happen with an SQL join at the server end. This turns out to be a bad idea though; you not only move the compute/I/O load up to the server, but to get any value out of doing that, you have to perform the query for a substantial number of records. If they have any number of relationships, that’s going to potentially impact your server performance and that’s something we don’t want to see with production servers.

Batch to the future

The solution was to batch requests in a smart way. Rather than request records one at a time, the team moved to request in larger (1000+) batches. For each relationship, a relationship record request is created by scanning that batch for the IDs of the relationships and crafting a query which would select the appropriate records for the entire batch. “Going back to our example, when we batch everything together the round trip time for the batch is the same as a query for a single record, and even if the query takes longer, 300 seconds becomes 3 seconds for a simple case, and 900 seconds becomes 10 seconds. It’s a huge difference.”

The queries being made on the server are also essentially simple queries so they should place minimal load on the server, unlike a complex join, and be safe to run alongside production workloads.

With this performance enhancement in place in 2021.5, we are looking forward to people being able to migrate their SQL data to MongoDB faster and from further afield than before.

Not just migrations

This isn’t the only performance optimization work going on in Studio 3T’s SQL stack either. Hugo explained that the team have been diving into various aspects of vendor’s SQL drivers and isolating issues within them too. Probably the most spectacular of those issues came with the Oracle driver. Its performance for import and migration was weak; in one test, importing two million records from Oracle was taking over four hours. In the forthcoming 2021.6 release of Studio 3T, that import took six minutes. It’s a huge boost and we aim to keep pushing the performance envelope on our import and export operations to SQL databases going forward.


How helpful was this article?
This article was hideous
This article was bad
This article was ok
This article was good
This article was great
Thank you for your feedback!

About The Author

Dj Walker-Morgan

Dj has been around since Commodore had Pets and Apples grew everywhere. With a background in Unix and development, he's been around the technology business writing code or writing content ever since.

Related articles

  • SQL Migration: MongoDB and SQL, Coexistence Made Easy
  • SQL Migration: MongoDB and SQL, Coexistence Made Easy
  • Moving from SQL to MongoDB? Studio 3T’s SQL Migration feature makes it simple
  • Test your skills: Using SQL in MongoDB Aggregation
  • Test your skills: Querying MongoDB with SQL

Tags

2022 academy aggregation AMA atlas Certification christmas community connections culture date tags events export features hackolade import intellishell In Use JSON knowledge base migration modelling mongodb mongoodb mongosh My 3T productivity product management query regex releases schema security SQL Studio 3T studio3t tasks time series tips updates webinar windows

Browse by MongoDB topic

  • Connecting to MongoDB
  • Database Administration & Security
  • Getting Started with MongoDB
  • Getting Started with Studio 3T
  • Import/Export
  • Job Automation & Scheduling
  • MongoDB Aggregation Framework
  • MongoDB/Studio 3T Workshops
  • Performance
  • Query & CRUD Operations
  • Reference
  • Schema
  • Studio 3T Licensing
  • Support and other resources
  • Working with MongoDB & SQL
  • Working with MongoDB Atlas

Studio 3T

MongoDB Enterprise Certified Technology PartnerSince 2014, 3T has been helping thousands of MongoDB developers and administrators with their everyday jobs by providing the finest MongoDB tools on the market. We guarantee the best compatibility with current and legacy releases of MongoDB, continue to deliver new features with every new software release, and provide high quality support.

Find us on FacebookFind us on TwitterFind us on YouTubeFind us on LinkedIn

Education

  • Free MongoDB Tutorials
  • Connect to MongoDB
  • Connect to MongoDB Atlas
  • Import Data to MongoDB
  • Export MongoDB Data
  • Build Aggregation Queries
  • Query MongoDB with SQL
  • Migrate from SQL to MongoDB

Resources

  • Feedback and Support
  • Sales Support
  • Knowledge Base
  • FAQ
  • Reports
  • White Papers
  • Testimonials
  • Discounts

Company

  • About Us
  • Blog
  • Careers
  • Legal
  • Press
  • Privacy Policy
  • EULA

© 2023 3T Software Labs Ltd. All rights reserved.

  • Privacy Policy
  • Cookie settings
  • Impressum

We value your privacy

With your consent, we and third-party providers use cookies and similar technologies on our website to analyse your use of our site for market research or advertising purposes ("analytics and marketing") and to provide you with additional functions (“functional”). This may result in the creation of pseudonymous usage profiles and the transfer of personal data to third countries, including the USA, which may have no adequate level of protection for the processing of personal data.

By clicking “Accept all”, you consent to the storage of cookies and the processing of personal data for these purposes, including any transfers to third countries. By clicking on “Decline all”, you do not give your consent and we will only store cookies that are necessary for our website. You can customize the cookies we store on your device or change your selection at any time - thus also revoking your consent with effect for the future - under “Manage Cookies”, or “Cookie Settings” at the bottom of the page. You can find further information in our Privacy Policy.
Accept all
Decline all
Manage cookies
✕

Privacy Preference Center

With your consent, we and third-party providers use cookies and similar technologies on our website to analyse your use of our site for market research or advertising purposes ("analytics and marketing") and to provide you with additional functions (“functional”). This may result in the creation of pseudonymous usage profiles and the transfer of personal data to third countries, including the USA, which may have no adequate level of protection for the processing of personal data. Please choose for which purposes you wish to give us your consent and store your preferences by clicking on “Accept selected”. You can find further information in our Privacy Policy.

Accept all cookies

Manage consent preferences

Essential cookies are strictly necessary to provide an online service such as our website or a service on our website which you have requested. The website or service will not work without them.

Performance cookies allow us to collect information such as number of visits and sources of traffic. This information is used in aggregate form to help us understand how our websites are being used, allowing us to improve both our website’s performance and your experience.

Google Analytics

Google Ads

Bing Ads

Facebook

LinkedIn

Quora

Hotjar

Reddit

Functional cookies collect information about your preferences and choices and make using the website a lot easier and more relevant. Without these cookies, some of the site functionality may not work as intended.

HubSpot

Social media cookies are cookies used to share user behaviour information with a third-party social media platform. They may consequently effect how social media sites present you with information in the future.

Accept selected
✕