System design
Why You Should Probably Not Use MongoDB
Sometimes I feel like Medium is filled with articles evangelizing the newest technology with titles like, “Why Rust Is The Best Language In Existence”, or “Ruby Is DEAD”. These articles often seem to serve the purpose of the authors’ anxieties in the modern job market more than anything else.
This is not one of those articles. I’ll never agree with the “this is better than that” mentality. The reality of software is that we need to pick the best tool for the job. That can be driven from business reasons such as using a JavaScript based stack due to it being ubiquitous across the stack and easy to hire for, or using Python because it has much more domain specific support for ML and scientific computing.
There does come a point in your career however when you start noticing that there are a few problematic parts of the stack that engineering teams quite often seem to lack the knowledge or experience to get it right. In my career this is definitely the case with MongoDB.
Document DBs Require A Paradigm Shift Rarely Taught
Relational databases are universally taught across university Computer Science and IT degree programs. Relational SQL however is almost universal. While normally I’d say the lack of formal education in a topic shouldn’t be an issue for engineers, the ubiquitousness of relational SQL paired with the lack of general training for Document DBs, in my experience seems to universally lead to poor outcomes outside of Fortune 500 companies with the resources to train engineering staff properly.
The shift in thinking from joining data together like you would in Excel to throwing all of your data in a hierarchal JSON document is a hard one and often the result of inexperienced teams or teams who lack technical oversight is using MongoDB in a relational way. Looking at the surface this doesn’t seem to be an issue, however, if we dive deeper it presents lots of tiny, hard to debug data integrity issues (See ACID below).
One tangible example of the paradigm shift is that in most relational setups, if we had an ERD where an organizations object had many users and users had many roles and so on, we’d just insert or delete a new role for example with a corresponding insert or delete statement. In a proper document setup, this looks like push and pop from an array which arguably, feels very foreign in the context of a database.
It also presents performance issues. Oftentimes writing MongoDB queries in a relational way yields several calls to the database OR poorly written and hard to optimize aggregations to do one thing which uses up connection pool resources leaving fewer connections available for concurrent requests which means fewer concurrent requests to your application before throwing more hardware at your MongoDB cluster.
ACID Compliance?
ACID is a mnemonic relating to a set of guidelines that databases follow to ensure your data is safe and the database is available for use at all times. A quick refresher:
Atomic
Larger procedural queries can be broken down into transactions and transactions are all-or-nothing. With atomicity you you’re protected against scenarios where you debit a bank customer’s entire balance and then delete their account but the debit fails and the account deletion succeeds which would lead to a phantom balance on the balance sheet.
Consistent
Transactions are available in to a read replica in whole chunks. That transaction is eventually replicated across all nodes of the database cluster exactly as it was written to the leader. See https://raft.github.io/ for more information.
Isolated
Concurrent transactions are run in isolation of each other. They don’t interfere with each other.
Durable
Once a transaction is committed and a network partition or loss of power occurs the data will be available upon restoration.
MongoDB is not ACID Compliant
Well, kind of. MongoDB is ACID compliant if you restructure your data to be document based. Meaning that if you have a schema along the lines of an organizations object owns an array of users which owns roles and so on then updating a single organization, or users or roles within an organization is ACID compliant.
Since MongoDB themselves realized that most people were using MongoDB relationally, in version 4 which was relatively late in it’s maturity, they introduced transactions. Now transactions don’t have first-class support in many Mongo ORMs. In many relational ORMs they’re so deeply baked in they’re a foolproof afterthought to any engineer using the ORM. This is definitely not the case with Mongo ORMs. Intentionality is the name of the game here which in fast moving engineering teams lacking experience is a recipe for tech debt and as any good CTO knows tech debt is a recipe for reduced velocity.
Even with transactions introduced, MongoDB may not be ACID compliant. Jepsen which does distributed systems safety research found that MongoDB transactions in version 4.2.6 we’re not actually ACID compliant despite the fact that they should in theory be so.
[## MongoDB 4.2.6
2020-05-26: MongoDB identified a bug in the transaction retry mechanism which they believe was responsible for the…
While this probably wouldn’t be an issue for small startups getting their start, it definitely might become an issue as that small startup begins to scale.
Phantom Fails
Since MongoDB is schema-less it lacks the immediate feedback querying that a database like PostgreSQL would provide. If you query Postgres with a misspelled variable name, the feedback is immediate and your console is painted in a red error message. Since MongoDB is schema-less the query will still complete but yield zero results. Features failing in this way often pass QA since no error is present. They may instead mark the feature as working or ask for a proper database seed. Later when this is found it’s usually at the expense of several engineering hours used to debug the feature.
While much of this can be avoided with duck-typing, TypeScript or just using a strongly typed language, many teams don’t. Even if you use TypeScript for example to eliminate many of these issues it still leaves aggregations, and several other more niche features vulnerable to this issue since ORMs don’t support typings in MongoDB for aggregations.
Aggregations
MongoDB offers aggregations in “pipeline stages” which is more or less what it sounds like. Imagining the pipe operator in bash and you’re halfway there. The difference here is that we’re not piping one file to another we’re doing an operation on each record of the collection which is computationally expensive.
Unless you have a match pipeline step to reduce your results before running another step on each document and optimize said match step with an index and place it as the very first pipeline step, you’re doing a table scan upon table scan on each document. This means that your app will decrease in performance as your collections with aggregations grow unless of course you were cognizant of writing an optimized match stage as your first pipeline step.
Replica Set For Change Feeds
Change Feeds or Triggers are a very useful feature. They can be used in real-time architectures such as live game updates or audit-logs. While Postgres, MySQL and others allow you to enable them with a simple ALTER statement, MongoDB only supports triggers when you’ve setup a replica set or a sharded cluster.
Personally, I was somewhat flabbergasted when I found this out. While most databases use their WAL (write-ahead log) to key off changes to the triggers / change feeds API, MongoDB doesn’t use it’s WAL (implemented via journaling in Wired Tiger) for triggers, it uses it’s OpLog which is used by the Raft Consensus Algorithm as the source of truth on the primary node before sending the change to all other replicated nodes and it’s adopted on each follower node’s OpLog.
While you can technically enable a single node replica set in MongoDB, in practice it’s poorly documented and rarely done. In our experience configuring this it seems MongoDB only supports connecting to a single node replica set with an obscure flag in the URI directConnection=true . The kicker here is that enabling a replica set in Docker environments, especially as a CI/CD pipeline for an integration testing environment is difficult.
For posterity, the solution we went with was a docker image pre-configured with a replica set, so we could avoid the rs.initiate() calls and the timing issues associated with it when running a CI/CD pipeline in the cloud.
The Rub
This article deserves to be boiled down into a simple problem and solution statement. The problem here is that MongoDB was envisioned as a schema-less document db to help teams iterate faster and provide a store for unstructured data. This caused teams to adopt it as a panacea of sorts, so MongoDB tried to adapt and introduced cheap patches but often fell a bit short.
The solution here is that MongoDB can work well only if you have an experienced MongoDB team. Experience here doesn’t mean that they’ve worked with MongoDB for 10 years. Anybody can work with MongoDB poorly for 10 years and that seems to be more common than not. MongoDB can only work well in an environment with technical leaders dedicated to using the technology properly and continuously audit the ERDs and schemas going forward. For all other teams that don’t want to or can’t spend the time monitoring the MongoDB implementation, Postgres or MySQL is the way to go.