Skip to content
Studio 3T - The professional GUI, IDE and client for MongoDB
  • Tools
    • Aggregation Editor
    • IntelliShell
    • Visual Query Builder
    • Export Wizard
    • Import Wizard
    • Query Code
    • SQL Query
    • Connect
    • Schema Explorer
    • Compare
    • SQL ⇔ MongoDB Migration
    • Data Masking
    • Task Scheduler
    • Reschema
    • More Tools and Features
  • Solutions
  • Resources
    • Knowledge Base
    • MongoDB Tutorials & Courses
    • Tool/Feature Documentation
    • Blog
    • Community
    • Testimonials
    • Whitepapers
    • Reports
  • Contact us
    • Contact
    • Sales Support
    • Feedback and Support
    • Careers
    • About Us
  • Store
    • Buy Now
    • Preferred Resellers
    • Team Pricing
  • Download
  • My 3T
search

Mongosh: How To Read More Files 

Posted on: 02/02/2023 by Dj Walker-Morgan

In this short series, we’re looking at the new capabilities of the IntelliShell and we start with how you can use Mongosh to read files.

Not all data is structured as JSON or CSV. A lot of valuable data is often trapped in plain text files on your disk. To get real value and insight from them, you will want to give the data some structure and turn it into a collection that you can query and aggregate in MongoDB. 

For example, if you had a plain text file of legacy data you wanted to use in MongoDB, previously you would have to think about writing an app to parse and import that data. But now, it’s possible to do that from within the Mongo shell and with Studio 3T’s IntelliShell.

A Change Of Shell

With the switch to default Mongosh in Studio 3T 2022.9, a whole world of new capabilities has been added to the Mongo Shell.

The old legacy mongo was a JavaScript interpreter attached to a database driver and a REPL (Read/Execute/Print/Loop) command line. There wasn’t much else to it. Also, it was hard to extend its JavaScript engine so that you could write files or connect to things that weren’t MongoDB. That meant you were more likely to have to write an actual program in your preferred language to do what you wanted.

With Mongosh, that’s all changed. The new shell is built on top of Node.js, the JavaScript runtime which is at the heart of many applications, both desktop and web. Because of this, we can harness the whole spectrum of modern JavaScript capabilities in our IntelliShell scripts. 

In this series, we’ll be exploring what you can do and why you should consider adding it to your toolbox. Today, we’ll talk about using Mongosh to read files. 

Getting Mongosh to Read Files

Why would you want to read files in the Mongo shell? Well, it’s a good way to provide input data, whether it’s commands or raw text which needs to be massaged into a JSON document so it can be inserted into a collection. In the older mongo shell, there was a command called “cat(<filename>)” that let you read a designated file into a variable, as a single string. 

With Mongosh, it officially replaces that command with:

fs.readFileSync( <filename>, 'utf8' )

This command reads an entire file in one go and it returns the contents as a string. The ‘utf8’ sets the character encoding. The “fs.” means it is part of the File System package in Node and if you look in the Node documentation for the fs package you’ll find it amongst a whole throng of versatile file reading and manipulating functions. 

We want to make things as simple as possible. Therefore we’re only going to focus on a couple of those functions, specifically the Sync() versions of the functions. These are functions which run and don’t need JavaScript callbacks or promises to work. If you know what they are you’ll know they can be a bit fiddly to work with. If you don’t, then we’re sticking with the simplest, most scripty version of the available functions. You call them, they do their work, they return a result. 

Now for a lot of uses, fs.readFileSync() will work because there’s plenty of memory to work with. 

Practical Mongosh File Reading

Let’s look at a practical example. We have a quotes file downloaded from github which contains a quote, a tilde and then the name of the person attributed to the quote. Sounds pretty simple, but it’s not in a format you can import. Let’s fix that and use it to create a collection.

Open IntelliShell in Studio 3T on a database or collection. We begin by reading our file:

const r=fs.readFileSync("/Users/dj/quotes.txt");

You’ll need to put the appropriate path into your copy of the quotes file. This function will read the whole file in one go. BUT, unlike the old shell’s cat(), it doesn’t read it into a string – instead, it reads it into a data structure called a Buffer. So, the next thing to do is turn that Buffer into a string:

const tmp=r.toString();

And once we’ve done that, we can use JavaScript’s string.split() function to break it up into an array of lines.

const sp=tmp.split("\n");

Excellent.

Writing to MongoDB

Of course getting Mongosh to read files is only half the story. Now we have to parse and store that data in MongoDB. We start that process by stepping through the array, line by line.

for (const s of sp) {

This is JavaScript’s automatic way of stepping through an array of things. Each line will appear as the constant s inside the loop:

  if(s=="") {
        continue;
    }

If s is a blank line, we skip the rest of the loop. Otherwise we move on to parsing the data. In this case it is simple. We split up the line on the tilde ~ character and save the left and right parts as quote and author:

   p=s.split("~");
    quote=p[0];
    author=p[1];

Now all we need to do is create a document and insert it into a collection.

    document={ "author":author,"quote":quote };
    db.getCollection("quotes").insertOne(document);
}

And yes, we are keeping it simple here. You could gather all the quote documents into one big JavaScript array and insert them at the same time with .insertMany(), but we’ll leave that as an exercise for you, the reader.

Here’s all the code in one chunk so you can easily copy and paste it:

const r=fs.readFileSync("/Users/dj/quotes.txt");
const tmp=r.toString();
const sp=tmp.split("\n");;

for (const s of sp) {
   if(s=="") {
        continue;
    }

    p=s.split("~");
    quote=p[0];
    author=p[1];

    document={ "author":author,"quote":quote };
    db.getCollection("quotes").insertOne(document);
}

Doing More With Mongosh

And there we have it. We’ve used IntelliShell and Mongosh to convert a plain text file with no formal structure into a MongoDB database. What you should take away from this is that your scripts now have the ability to access anything that a Nodejs application might want. In future, we’ll look at even more capabilities – writing files and launching other processes – all from within Mongosh.


How helpful was this article?
This article was hideous
This article was bad
This article was ok
This article was good
This article was great
Thank you for your feedback!

About The Author

Dj Walker-Morgan

Dj has been around since Commodore had Pets and Apples grew everywhere. With a background in Unix and development, he's been around the technology business writing code or writing content ever since.

Related articles

  • Mongosh: How To Write More Files
  • MongoDB Atlas, Read Preference Tags and Studio 3T
  • Mongosh: How to Run More Processes
  • How do you make Studio 3T export MongoDB collections to unique files? #Studio3T_AMA
  • Configure mongodump, mongorestore & mongo shell Executable Files

Tags

2022 academy aggregation AMA atlas Certification christmas community connections culture date tags events export features hackolade import intellishell In Use JSON knowledge base migration modelling mongodb mongoodb mongosh My 3T productivity query regex releases schema security SQL Studio 3T tasks time series tips updates webinar windows

Browse by MongoDB topic

  • Connecting to MongoDB
  • Database Administration & Security
  • Getting Started with MongoDB
  • Getting Started with Studio 3T
  • Import/Export
  • Job Automation & Scheduling
  • MongoDB Aggregation Framework
  • MongoDB/Studio 3T Workshops
  • Performance
  • Query & CRUD Operations
  • Reference
  • Schema
  • Studio 3T Licensing
  • Support and other resources
  • Working with MongoDB & SQL
  • Working with MongoDB Atlas

Studio 3T

MongoDB Enterprise Certified Technology PartnerSince 2014, 3T has been helping thousands of MongoDB developers and administrators with their everyday jobs by providing the finest MongoDB tools on the market. We guarantee the best compatibility with current and legacy releases of MongoDB, continue to deliver new features with every new software release, and provide high quality support.

Find us on FacebookFind us on TwitterFind us on YouTubeFind us on LinkedIn

Education

  • Free MongoDB Tutorials
  • Connect to MongoDB
  • Connect to MongoDB Atlas
  • Import Data to MongoDB
  • Export MongoDB Data
  • Build Aggregation Queries
  • Query MongoDB with SQL
  • Migrate from SQL to MongoDB

Resources

  • Feedback and Support
  • Sales Support
  • Knowledge Base
  • FAQ
  • Reports
  • White Papers
  • Testimonials
  • Discounts

Company

  • About Us
  • Blog
  • Careers
  • Legal
  • Press
  • Privacy Policy
  • EULA

© 2023 3T Software Labs Ltd. All rights reserved.

  • Privacy Policy
  • Cookie settings
  • Impressum

We value your privacy

With your consent, we and third-party providers use cookies and similar technologies on our website to analyse your use of our site for market research or advertising purposes ("analytics and marketing") and to provide you with additional functions (“functional”). This may result in the creation of pseudonymous usage profiles and the transfer of personal data to third countries, including the USA, which may have no adequate level of protection for the processing of personal data.

By clicking “Accept all”, you consent to the storage of cookies and the processing of personal data for these purposes, including any transfers to third countries. By clicking on “Decline all”, you do not give your consent and we will only store cookies that are necessary for our website. You can customize the cookies we store on your device or change your selection at any time - thus also revoking your consent with effect for the future - under “Manage Cookies”, or “Cookie Settings” at the bottom of the page. You can find further information in our Privacy Policy.
Accept all
Decline all
Manage cookies
✕

Privacy Preference Center

With your consent, we and third-party providers use cookies and similar technologies on our website to analyse your use of our site for market research or advertising purposes ("analytics and marketing") and to provide you with additional functions (“functional”). This may result in the creation of pseudonymous usage profiles and the transfer of personal data to third countries, including the USA, which may have no adequate level of protection for the processing of personal data. Please choose for which purposes you wish to give us your consent and store your preferences by clicking on “Accept selected”. You can find further information in our Privacy Policy.

Accept all cookies

Manage consent preferences

Essential cookies are strictly necessary to provide an online service such as our website or a service on our website which you have requested. The website or service will not work without them.

Performance cookies allow us to collect information such as number of visits and sources of traffic. This information is used in aggregate form to help us understand how our websites are being used, allowing us to improve both our website’s performance and your experience.

Google Analytics

Google Ads

Bing Ads

Facebook

LinkedIn

Quora

Hotjar

Reddit

Functional cookies collect information about your preferences and choices and make using the website a lot easier and more relevant. Without these cookies, some of the site functionality may not work as intended.

HubSpot

Social media cookies are cookies used to share user behaviour information with a third-party social media platform. They may consequently effect how social media sites present you with information in the future.

Accept selected