Distributed computing

Why CockroachDB Is So Hard to Kill

Raft is only the start. Leases, closed timestamps, hybrid-logical clocks and self-healing replicas are what keep CockroachDB alive when hardware dies.

In the United States we have a fatalistic streak, with jokes about what would survive a nuclear war. There are generally only two punchlines: Twinkies and cockroaches. So it’s no wonder CockroachDB is named the way it is. In January 2012, before a line of code existed, Spencer Kimball emailed a friend that the tentative name for his new database would be “cockroach”: “very fast, very scalable, very hard to kill” (Cockroach Labs). Cockroach Labs still describes it as a database that “survives disk, machine, rack, and even datacenter failures” (FAQ).

How does one actually achieve that? Surviving disk, rack and datacenter failures isn’t a new idea. It’s baked into managed services like RDS, usually as a primary with standbys that take over when it fails. The hard part is doing it with minimal latency disruption, without a failover window, and without giving up consistency. That’s the real trick, and this article walks through how CockroachDB pulls it off, using CockroachDB’s own architecture documentation as the source.

The short version is that CockroachDB treats failure as the normal case. It doesn’t have one copy of the truth that it protects. It has many small, independently replicated pieces of the truth, each of which keeps working as long as most of its copies can still talk to each other, and a set of mechanisms layered on top of consensus so that surviving doesn’t cost you speed.

Ranges

When we talk about recovering from datacenter failures, we’re talking about data replication. In a single-node database, the database locks the tables or rows it needs and queues requests while a transaction completes. There’s one copy of the data and one authority over it. If that machine dies, the data is unavailable until a standby takes over, and any write that hadn’t reached the standby is a question mark.

CockroachDB starts by refusing to treat the database as one thing. Every table, index and system record is mapped into one sorted key space, and that key space is split into contiguous chunks called ranges (distribution layer). By default a range splits when it reaches 512 MiB and can merge with its neighbor when it falls below 128 MiB (replication zones). Finding a key is a two-level lookup. A small meta1 range, which every node knows how to find and which never splits, points to meta2, which points to the range holding the data, and nodes cache those lookups as they go.

The unit of survival is the range, not the database. A cluster with a few terabytes of data has thousands of ranges, each replicated, each with its own leader, spread across every node. Losing a node doesn’t take down “the database.” It takes away one copy of a few thousand ranges, each of which still has its other copies.

Three nodes each hold a replica of ranges A, B and C. Node 1 holds the lease for A, node 2 for B, node 3 for C. — Every range is its own Raft group, replicated across nodes, and the leases are spread out, so every node leads some ranges and follows others. Simplified to three ranges; a real cluster has thousands.

Consensus Per Range

Each range’s replicas form their own Raft group (replication layer). One replica is the leader, and a write commits only when a majority of the group has appended it to its log. With the default of three replicas, that’s two of three.

A client sends a write to the leader, which appends it to two followers; once one follower acknowledges, a majority of two of three has it and the write is committed. — A write in Raft: the leader appends it to the followers’ logs and commits once a majority has it.

That majority rule is where the survival math comes from. A range with $n$ voting replicas keeps working through $f$ simultaneous failures, where

f = \left\lfloor \frac{n - 1}{2} \right\rfloor

Three replicas survive one failure, and five survive two. When a leader dies, the surviving replicas hold an election and pick a new one, with a randomized timeout so they don’t all stand for election at once. Every node can lead some ranges and follow others, so there’s no primary node to lose. Each node is a leader for some of the data and a backup for the rest.

Pure Raft has a cost, though. In textbook Raft, even a read has to go through the leader and confirm it’s still the leader, which is another round trip. If that were the whole design, survivability would come with a latency tax on every query. CockroachDB adds several layers on top of Raft so that it doesn’t.

A client asks the leader to read x; the leader first asks a follower whether it is still the leader, gets a yes, then answers. — A read in textbook Raft: the leader confirms it’s still the leader before answering, one more round trip for every read.

Leases: Raft Without the Round Trip

The first layer is the range lease. One replica in each range holds the lease and is the only one allowed to serve strongly consistent reads or propose writes. Because the lease guarantees that the holder has seen every committed write for its term, it can answer a read from local state without asking anyone (replication layer).

A client asks the leaseholder to read x and gets the answer straight back from local state. — The same read in CockroachDB: the lease is the leaseholder’s proof that it has every committed write, so it answers without asking anyone.

Since v25.2, CockroachDB uses what it calls leader leases, which keep the lease with the Raft leader except briefly during a transfer, so the leaseholder can propose commands to itself instead of forwarding them. Failure detection runs through a mechanism called store liveness, and when a leaseholder dies, a new Raft leader is elected and takes the lease, which the docs say should complete “within a few seconds.” For that range, that’s the whole failover.

Reading From Anywhere

So what happens to a read when the leaseholder is on another continent? The second layer is about reads that don’t need to be the absolute latest. The leaseholder continuously advances a closed timestamp for its range, a promise that no new write will ever land at or below that time. Any replica that has caught up to the closed timestamp can serve reads at or below it, so a read can go to the nearest replica instead of crossing an ocean to the leaseholder (follower reads).

These are exposed in SQL as AS OF SYSTEM TIME queries. Exact staleness reads run at a fixed time in the past, and the docs say they should tolerate data at least 4.2 seconds old. Bounded staleness reads let the database pick the freshest timestamp it can serve locally, and they keep working through network partitions.

The third layer is non-voting replicas, added in v21.1. They follow the Raft log and can serve those follower reads, but they don’t count toward the majority. You can put a copy of a range in a distant region for fast local reads without making every write wait on a cross-ocean vote. You set this with num_voters lower than num_replicas in a replication zone.

Clocks Without Atomic Clocks

Every distributed database has to agree on the order of events, and the hardest version of that problem is time. Google’s Spanner solves it with TrueTime, GPS receivers and atomic clocks in its data centers that keep clock uncertainty “generally less than 10ms” (Spanner). CockroachDB runs on commodity hardware, so it can’t assume that.

Instead it uses hybrid-logical clocks: a physical component that stays close to each node’s wall clock, plus a logical counter that orders events sharing the same physical time (transaction layer). It assumes clocks are within a maximum offset of each other, 500 ms by default, and it enforces that assumption violently. A node whose clock drifts from at least half of the cluster by 80% of the maximum offset, 400 ms by default, shuts itself down (operational FAQ). That sounds drastic, but it’s the right call. A node with a bad clock is far more dangerous running than dead, because the rest of the cluster can survive its absence.

Within that bound, a read treats any value written slightly after its own timestamp as uncertain, since a clock could be off by that much. Rather than guessing, the transaction pushes its own timestamp past the uncertain value and continues. The docs note that a retried transaction never runs into uncertainty again on a node it has already visited.

The trade-off is documented and worth knowing. When Jepsen tested CockroachDB in 2017, it found two serializability violations, which were fixed, and confirmed that CockroachDB provides serializability but not strict serializability. Transactions on unrelated keys can appear in a different order than they happened in real time. Where Spanner waits out its clock uncertainty after every write, CockroachDB blocks only on contested reads. Jepsen also confirmed that loose clock synchronization is necessary for correctness, which is exactly why nodes kill themselves when they lose it.

Transactions That Don’t Pay Twice

Consensus makes every write slower than a single-node write, so the transaction layer is built to pay that cost as few times as possible. Writes go down first as provisional values called write intents, each pointing at a transaction record that says whether the transaction is pending, committed or aborted. Any other transaction that runs into an intent can check the record and resolve it, so a coordinator that dies mid-transaction doesn’t leave locks stranded forever.

Two optimizations keep that from being slow. Transaction pipelining lets a transaction keep issuing writes without waiting for each one to finish consensus, and settles all of that at commit. Parallel commits then let the commit itself happen in parallel with the final writes, which the docs say “cuts the commit latency of a transaction in half, from two rounds of consensus down to one.” All of this happens at SERIALIZABLE, which is CockroachDB’s default and the strongest ANSI isolation level.

Healing

Surviving a failure is only half of being hard to kill. The other half is getting back to full strength without anyone touching it. When a node stops updating its liveness record for server.time_until_store_dead, five minutes by default, CockroachDB considers it dead and starts rebuilding its replicas on healthy nodes (cluster overview). A leaseholder sends a snapshot of the range to the new replica, which then replays the Raft log to catch up. Add a node, and the cluster rebalances ranges onto it. No one reshuffles data by hand.

Why wait five minutes? The delay is deliberate. A node that’s rebooting for a patch will be back in less than that, and copying hundreds of gigabytes for a node that’s about to return would cost more than it saves. During those five minutes every affected range is still serving traffic from its other replicas, just with less margin for a second failure.

Surviving a Region

Everything above protects against losing nodes. CockroachDB’s multi-region features extend the same idea to losing a whole availability zone or region, and make you choose what that costs (survival goals). The default survival goal is zone failure: the database stays fully available for reads and writes if a zone goes down. Region survival requires at least three regions and raises the replication factor from three to five, spread two, two and one, so a majority survives losing any one region. What does that cost? Every write has to reach a second region, which adds at least the round-trip time to the nearest region. Reads are unaffected.

For read-mostly data that every region needs, like a product catalog, CockroachDB has GLOBAL tables. Writes to them are scheduled slightly in the future, about 800 ms with default settings, and wait for real time to catch up before acknowledging (Cockroach Labs). Because the range has promised in advance that nothing will be written before that future time, every replica in every region can serve current, consistent reads locally (global tables). Writes get slower so that reads anywhere in the world get fast.

What It Won’t Survive

CockroachDB is a CP system. If a range loses its majority, it stops accepting writes for that range rather than risk two sides of a partition disagreeing. If you run three replicas and lose two, that range is unavailable until one comes back. Being hard to kill depends on placing replicas so that one event can’t take a majority, which is what replication zones and survival goals are for.

It also depends on clocks. The design assumes clock offsets stay inside the configured maximum, and the self-termination check protects it, but Jepsen noted that monitoring can’t always react to a large, sudden clock jump. And for losing an entire cluster or a cloud provider, the architecture doesn’t help. That’s what backups and cross-cluster replication are for.

Takeaways

If I were designing a system that had to stay up through hardware failures, I’d take three lessons from CockroachDB. Make the unit of failure small. Thousands of independently replicated ranges degrade gracefully, while one big primary fails all at once. Put consensus underneath, then spend most of your engineering on not paying for it on every request: leases for strong reads, closed timestamps for nearby reads, non-voting replicas for distant ones, and pipelined, parallel commits for writes. And when a component can’t prove it’s healthy, like a node whose clock has drifted, have it remove itself. A system can route around something that’s gone. What it can’t route around is something that’s still there and wrong.