Backend Architecture for Moving Fast

Oftentimes your ability to create and sell a new product, or retain your existing users, depends on how fast you can ship new features. This is in fact the entire central point behind agile; to ship features quickly. Agile does us no good, however, if things have ground to a halt on the development side of things. Much of the time backend development grinding to a halt, or really any development slowing to a halt has more to do with a lack of systems than anything else.

“You do not rise to the level of your goals. You fall to the level of your systems.” — James Clear in Atomic Habits

In our case, systems could be our tools, our CI/CD, or a set of principles our team can follow to help identify issues with velocity. Most of the issues listed in this article are common backend issues I’ve noticed working and consulting for a number of startups. These observations all violate one of five rules that I use to determine if it will help us move faster. Those are:

1. Maintainability — Does this change make our codebase or processes more maintainable? If so, time to ship will drop and quality will go up having a compounded effect on velocity.
2. Observability — Does this change make something in the stack more observable, bringing the change front and center where it wasn’t before? If so, expect a sizable increase in velocity due to cutting down R&D time.
3. Predictability — Does this make some part of our process or codebase more predictable? If so, this will yield a fairly significant increase in velocity due to reducing decision fatigue.
4. Transparency — Does this change bring transparency, either to a part of the codebase or a part of our process where there wasn’t enough transparency before? If so, this cuts unneeded communication.
5. Automation — Does this change automate behavior for us? If so, this is a 1:1 time saver.

When creating a project, debates and decision fatigue often take over the bootstrapping stages slowing progress down significantly. One way to alleviate this and provide long-term maintainability, transparency and predictability is using a well-documented opinionated framework. This will help your team reduce decision fatigue by shipping code that closely adheres to the framework’s documentation.

Since setting up the scaffolding is arguably one of the most daunting and time-consuming tasks, innumerable boilerplate repositories and paid services have popped up in the past few years to help teams streamline bootstrapping. While I don’t recommend those solutions because the open-source repositories often aren’t kept up to date or they’re too opinionated, some frameworks that you can try depending on your stack are:

• Java — Spring Boot
• Ruby — Ruby on Rails
• Node.js — Nest.js
• Python — Django
• Rust — Axum
• Go — Beego

Some of these are more opinionated than others. The more opinionated a framework is, however, the harder thinking outside the box becomes, so balance is key here.

2. Ports & Adapters

The Ports & Adapters pattern, also known as Hexagonal Architecture, promotes loose coupling and easier refactoring. By using interfaces and dependency injection, you can swap out implementations without affecting the core business logic.

This approach makes it easy to add new services or switch between implementations.

3. Use Contracts

In interpreter-land, this is probably one of the bigger mistakes. APIs need a strict contract with their consumers. If the producer of a message and the consumer of a message isn’t held to the same contract, avoidable bugs will start to spawn with frequency.

GraphQL is one of my favorite ways to avoid this problem since once you define your GraphQL typings, your “routes” are generated with your request type and response type and GraphQL libraries generally generate all of the types from your GQL type definitions, regardless of which language you’re using.

ProtoBuf is nice too but becomes a bit cumbersome without the knowledge about what constitutes a breaking change and the CI/CD tools to detect them.

If all else fails, type packages can be published privately and downloaded to all of your producers and consumers so your compilers across the stack can enforce those types. Shout-out to Go for making this especially easy.

4. Make Intelligent Use of Middleware

Middleware is incredibly powerful and in my opinion, quite often underused. No matter if you’re building an audit log, or implementing authentication, middleware does the job.

Middleware should be leveraged anywhere you may repeat code. One of the more clever uses of middleware that has saved us quite a bit of time, is to create different error classes that extend our language’s native error class based on different HTTP codes.

While I wrote this example to show you how this works, there are libraries that provide these error classes already made for you so all you have to do is write the filter to catch the error and send it back. While it requires a bit of setup, this keeps your code free of error handling in a way that makes sense for most teams, and your messages are uniform and pre-formatted.

This obviously isn’t the only way to use middleware. Combining middleware with database triggers for an audit-log is another decent use for middleware, as is response formatting.

5. Rely On Contract & DB Validation as Much As Possible

When pairing contract-based (specifically such as a decorator, tag, or macro based validator) with filtering, and sprinkling in just enough unique indexes, you’ve got yourself a streamlined and maintainable input, error and validation stack that’s incredibly powerful and helps increase the transparency and maintainability of your codebase by putting validations side-by-side with the contract it’s validating instead of separating the two.

As you can see, with decorators, where we define our contract, we can also define our validations. This reduces friction in code reviews making it much easier for the reviewer to inspect validations and for the reviewee, it becomes almost muscle memory writing validations as you’re writing your contract.

While Java and Ruby both have similar functionality and libraries, Rust has a few macro based validators as does Go have a few tag based validators.

6. Use a Relational Database

NoSQL databases often lead to significant slowdowns. The schema-less design loses points in predictability and maintainability because the lack of a strict contract causes silent failures.

In Postgres for example, when you write a query with a column that doesn’t exist or an aggregation with a table that doesn’t exist, Postgres will send back an error message immediately, with NoSQL databases they will silently fail and return an empty array indicating that there are no results because in theory that table or column could exist since the database has no self-awareness. This means hours to find that missing s in your aggregation pipeline or one of your filters.

7. Use a Strict Style Guide

Style guides are great when it comes to linting. They often catch errors that junior developers often make, and keep seniors writing clean and performant code preventing the many “gotchas” that languages and frameworks can have. This provides an improvement of predictability which translates to an improvement in velocity.

8. Observability First

While tools like Sentry, New Relic and DataDog are often seen as a cost center — not as a net benefit for the team, they pay off in dividends when it comes to velocity. REPL cycles for debugging with console logs that need to be deliberately placed takes several orders of magnitude longer than just having proper logging and observability from the beginning. Likewise with performance issues, methodically placing console.time() and console.timeEnd() throughout every layer of your app, deploying, and then pouring through the logs without search, is hours slower than having that data in a dashboard in front of you. Observability from day one is incredibly important.

9. Require Code Owners

Code reviews are quite often a bottleneck. Not requiring a code owner can cause reviewers and their requirements to change, removing predictability for the developer, however keeping a single reviewer may cause issues to be missed. The best way around this is to:

• designate a primary reviewer for each component
• rotate secondary reviewers to provide fresh perspectives
• maintain a living document or weekly slack post to call out common review comments to reduce repetition

Will Larson who wrote Staff Engineer: Leadership Beyond The Management Track, says in the book that architectural vision should be done by one person. While in my opinion, this doesn’t mean one person for the entire organization, it definitely can mean different SMEs for different parts of the stack and the app.

10. Use a Local CI/CD (Git Hooks)

There have been jokes floating around for ages on how and when to commit. One joke goes, “Committing should take place like voting in Chicago; early and often”. While I have an opinion much more nuanced than that, for absolute simplicity’s sake, this isn’t a bad rule of thumb, especially for large problematic projects that require going through git history.

Since this is a fairly common practice however, oftentimes engineers introduce new bugs while committing frequently. Even worse, quite often engineers push code that fails the linter or has failing unit tests.

Tools like Husky which expand the functionality of Git Hooks, allow us to run linting, validate our commit messages against the conventional commit regex allowing us to automate semantic versioning, and ensure our unit tests pass before allowing the code to be pushed.

Due to this, Git Hooks can significantly decrease our time running linters manually, waiting for CI/CD to run in the cloud, speed up our semantic versioning process by allowing us to automate it, and generally provide us more maintainable code and better commit messages that in theory can help us bring down our debugging time in the future.

Outro

While implementing these practices requires upfront investment, they yield significant returns in development velocity. By focusing on maintainability, observability, predictability, and transparency, you create an environment where your team can move lightning fast without sacrificing quality.

Remember, the goal is not just to move quickly, but to sustain that speed over time. These architectural practices provide the foundation for long-term velocity in backend development.