Elasticsearch and MongoDB are the two most popular distributed datastores used to manage NoSQL data. Both of these technologies are highly scalable and have document-oriented design at the core. There are differences between the two technologies, however, and it’s important to understand these differences in order to choose the right one for your use case. This blog post will examine the differences between these two technologies in a number of critical areas.

About Elasticsearch

Elasticsearch is an open-source, distributed RESTful search engine written in Java. It is built on top of Apache Lucene and extends Lucene’s functionality with HTTP web interface and data distribution using the index and shards concept. An index in Elasticsearch is similar to a database. It organizes data under a namespace, has a defined schema, and can be divided into multiple shards for horizontal scaling. Each record in Elasticsearch is stored as a JSON object and is called a “document.”

Some of the core features of Elasticsearch include:

  • Distributed search
  • High availability
  • REST interface
  • Powerful query DSL
  • Multitenancy
  • Geo search
  • Horizontal scaling

Despite having a rich list of feature sets, Elasticsearch is not the perfect data store for all scenarios. It has a few limitations that need to be taken into account when choosing the right data store for your application.

About MongoDB

MongoDB is a document-oriented database written in C++ and designed to handle terabytes of data spread across multiple geographic locations. In MongoDB, you can create multiple databases, and each database can have multiple collections (tables). As with Elasticsearch, each record in MongoDB is stored as a JSON object and called a “document.” MongoDB is a schemaless database that supports great built-in security features such as authentication, access control, and encryption. 

Some of the core features of MongoDB are:

  • Distributed document storage
  • High availability
  • Schemaless
  • Powerful queries and aggregations
  • Horizontal scaling
  • Built-in security
  • Great indexing capabilities
  • Geo search
  • GridFS to store any size document

The biggest limitations of MongoDB are its inability to provide full-text search at speed and its lack of some search functionalities, such as text tokenization.

Elasticsearch vs. MongoDB: A Detailed Comparison

As illustrated above, these technologies have a lot of similarities in their designs and features. That said, they differ greatly in nature. Elasticsearch is primarily a search server, while MongoDB is primarily a database. Let’s look at the differences between them in other areas.

Use Cases

Your use case will be critical in deciding which technology is the perfect fit. Elasticsearch will always be the better choice when full-text search is a requirement. Elasticsearch also wins the race when it comes to log analytics, since not only does it offer a wide range of aggregation queries, it also supports products like Kibana, Logstash, and beats—all of which make log analysis much easier.

On the other hand, when the data is in NoSQL format and you need a highly scalable database which requires CRUD operations without full-text search support, MongoDB is a reliable choice. MongoDB also supports full-text queries with the help of text-based indexes, but its search speed is slow and it lacks the tokenizers and analyzers that come with a search server

Configurations Files

The installation package of both Elasticsearch and MongoDB are available under all flavour of Linux, windows and Mac operating systems. Once you install the package, the default configurations are good to start with, but here are some important configuration parameters that you should modify before taking them into production. All the following configuration options are shown as per Linux Operating System. 

The configuration files for elasticsearch are stored under /etc/elasticsearch/config directory as shown below:

config
|-- elasticsearch.keystore
|-- elasticsearch.yml
|-- jvm.options
|-- log4j2.properties
|-- role_mapping.yml
|-- roles.yml
|-- users
`-- users_roles

Where, all the mongodb configurations can be done inside only a single file which is found under /etc/mongod.conf

Backup Recovery

Both Elasticsearch and MongoDB offer backup and recovery functionality by default. 

Elasticsearch performs incremental backups using _snapshot REST endpoint with the help of plugins, and its backup destinations can vary from file systems to cloud storage. The benefit of snapshots is that they are incremental in nature. Old snapshots can be deleted easily, and the recovery of snapshots is super easy to configure. The snapshot API does not offer queryable backup, however.

For example,

To take the Elasticsearch backup in S3 buckets, you need to install the S3 repository plugin on every Elasticsearch node, using following command:

sudo bin/elasticsearch-plugin install repository-s3

And then register a repository inside your AWS s3 bucket, 

curl -X PUT "localhost:9200/_snapshot/test_s3_repository?pretty" -H 'Content-Type: application/json' -d'

{
  "type": "s3",
  "settings": {
    "bucket": "s3_bucket_name"
  }
}  

Once repository is registered, you can start taking snapshots using following command:

curl -X PUT "localhost:9200/_snapshot/test_s3_repository/snapshot_1?pretty"

MongoDB offers multiple ways to perform backups. The first is the “mongodump” tool, which ships with the MongoDB installation and is the most common solution DevOps teams use. While mongodump has some limitations—it does not take incremental backups and is not effective for large databases—it offers features like 1) queryable backup, 2) whole database backup, and 3) individual collection. 

To implement incremental backup in MongoDB, you need to use the MongoDB oplog, which is a capped collection. It is also possible to create a backup of a MongoDB deployment by taking a snapshot of the file system. This makes a copy of MongoDB’s underlying data files. MongoDB’s Enterprise edition allows you to access other options, such as MongoDB Atlas, MongoDB Cloud Manager, and MongoDB Ops Manager.

To take the backup using mongodump, you just need to run following command:

mongodump --db <database_name> --host <mongohost_ip_address>

But, unlike Elasticsearch snapshots, mongo dumps will save on local disk and not into the S3 buckets or any other cloud storage.

Popularity

According to DB-Engines, Elasticsearch has the number one rank in search engines and seventh overall. MongoDB takes the number one spot in document store databases and fifth overall.

Figure 1: DB-Engines Ranking—Elasticsearch vs. MongoDB Popularity (Source: DB-Engines)

Support for Handling Relational Data

NoSQL data stores are built for scaling, high throughput for writing, and reading queries, but they are unable to handle relational data and do not possess the ACID properties offered by relational databases. The data in relational databases are stored in rows and columns. While data can be normalized easily, Elasticsearch and MongoDB support the document model and therefore focus on keeping data in a denormalized format. 

While there is no hard and fast rule for data modeling in these data stores, it is customary to either rely on keeping duplicate data in documents or perform application-side joins.

Despite its limitations, Elasticsearch has two built-in functionalities for handling relational data: the 1) nested and 2) parent-child document models. 

MongoDB also has two ways to handle relational data. One is the embedded document model, in which related objects enter storage as subdocuments. The other is the reference model, which includes links or references from one document to another.

Data Storage Architecture: Lucene vs. C++

Elasticsearch is built on top of Lucene and uses Lucene segments to write data inside inverted indexes. The metadata information—such as index mapping, settings, and other cluster states—is written in Elasticsearch files on top of Lucene. 

The problem with Lucene segments is they are immutable in nature and each commit creates a new segment. These segments merge behind the scenes based on merge settings. This makes data updates into heavy operations because, when each document updates in place, it is deleted and a new document is generates in its place. 

To avoid generating too many segments and significant I/O, Elasticsearch maintains a transactional log for each index, avoiding a low-level Lucene commit on each indexing operation. Transaction logs are also useful for recovering data in case of a crash or data corruption event.

MongoDB’s underlying storage model is completely different from Elasticsearch’s. MongoDB is written in C++ and uses a memory map file to map an on-disk data file to an in-memory byte array. It uses a doubly linked list data structure to organize the data. Each document contains a linked list to every other document as well as to the actual BSON-encoded data under the hood. MongoDB uses journal logs to help with database recovery in case of a hard shutdown. Eventually, the MongoDB process will be killed if there is low system memory or very high utilization of other system resources.

These differences demonstrate that MongoDB is built for 1) high write and 2) update throughput without causing high CPU and disk I/O issues. 

Document Size

The default maximum document size that Elasticsearch supports only goes up to 100 MB, although you can increase this maximum to 2GB—Lucene’s limit. However, it’s important to keep in mind that very large documents often create additional issues.

By default, MongoDB supports the storage of documents up to 16 MB. You can store larger documents using GridFS functionality.

Licensing Model, Monitoring, and Security

Elasticsearch is an open source tool that comes with an Apache 2.0 license. The open-source distribution has all of the functionalities you need for building a search application along with basic security features. For alerting and monitoring, you can opt for various open source plugins. If you need additional functionalities such as advanced security, alerting, and machine learning, you will have to purchase a subscription to the Gold, Platinum, or Enterprise versions of Elasticsearch.

MongoDB is also free to use, and its community edition comes with a Server Side Public License (SSPL) v1.0. The community edition contains all of the core MongoDB features, such as basic monitoring tools and security. If you plan to explore and use advanced features such as MongoDB Management, Advanced Monitoring, In-Memory database engine, and BI-Connector, you can opt for the MongoDB Enterprise edition.

It’s also possible to monitor MongoDB using the ELK Stack, which you can read more about in this tutorial.

Programming Language: Java vs. Lucene

Elasticsearch is written in Java, and MongoDB is written in Lucene; however, both of these technologies offer a wide range of client support in multiple languages. Elasticsearch has clients available for Java, Javascript, Ruby, GO, .NET, PHP, Perl, Python, and Rust. In addition, there are several community-contributed clients available for languages such as C++, Scala, and R.

MongoDB offers an even wider range of drivers for other programming languages such as C, C++, Scala, and Swift. There are multiple community-contributed clients available for MongoDB as well.

Summary

Both Elasticsearch and MongoDB are built for specific use cases, but there may be common scenarios where the choice of one tool over the other may be more complex. In this blog, we have reviewed and compared various features of both technologies in order to help you make these more difficult decisions. 

To summarize, MongoDB is a very popular and scalable NoSQL database that is a leader in document-oriented databases. It is usually the best solution when the use case necessitates a highly scalable database with high throughput transactions. When it comes to handling full-text search, log analytics, finding anomalies, and root cause detection, Elasticsearch is the clear winner.