Distributed computing

Real-Time Notifications, a Lesson in CAP

We wanted real-time Anaheim Ducks game notifications from a feed that serves the whole internet. A retracted goal looked exactly like a stale read, so we measured how long the feed took to settle and turned “real time” into a number.

At OC Sports + Entertainment, which is owned by the Anaheim Ducks, we wanted real-time game notifications. A goal is scored at the Honda Center and a fan’s phone lights up. Penalties, period ends, the final horn. It sounds like the simplest feature in sports software. Watch the game data, and when something changes, tell people.

The data came from the NHL, and the NHL serves its game data to the entire internet. Every app, every broadcaster’s graphics, every fantasy site and every fan refreshing a box score reads from the same feed. A feed with that kind of audience doesn’t serve one copy of the truth. It serves cached copies, from servers all over the world, each a little older or newer than the last.

That’s where it went wrong. When we polled the feed and a goal disappeared, there were two possible explanations. Maybe the goal was really taken back, overturned on review or reassigned. Or maybe we’d just hit an older copy of the data, and the goal would be back on the next poll. From the outside, those looked the same. A real retraction and a stale read were indistinguishable, and the difference between them is the CAP theorem.

What CAP Actually Says

In 2000, Eric Brewer conjectured that a distributed system can’t have consistency, availability and partition tolerance all at once, and in 2002, Seth Gilbert and Nancy Lynch proved a formal version of it (Gilbert and Lynch). It’s usually taught as “pick two of three,” and that framing has done a lot of damage.

The words mean something narrower than they sound. As Martin Kleppmann points out, “consistency in CAP actually means linearizability,” a very strong guarantee that every read sees the latest write, and availability means every request to a working node gets a real answer (Kleppmann). And partition tolerance isn’t really a choice. Networks fail, so a distributed system has to decide what to do when they do. The actual trade-off is narrower: when parts of the system can’t talk to each other, do you refuse to answer until you’re sure, or answer with what you have?

Here’s why there’s no third option. Two copies of the score, one in Los Angeles and one in New York. A goal is recorded in Los Angeles, and before the update reaches New York, the network between them fails. A fan in New York asks for the score. Play it both ways:

Cut the network Wait until sure (consistency) Answer anyway (availability)

Press Play. A goal is written in Los Angeles, and a fan in New York asks for the score.

An illustration of the trade-off at the heart of the CAP theorem. Without a partition, a system can be consistent and available at once. During one, the replica that missed the write has to choose.

Replica B can’t return the new score, because it never received it, and it can’t know whether a newer score exists, because the only way to find out is the link that’s down. So it either waits, and the fan gets nothing, or it answers, and the fan might get something old. Consistency or availability. The partition doesn’t let it have both, and it doesn’t ask permission to happen.

Brewer said as much himself twelve years later. “The ‘2 of 3’ formulation was always misleading because it tended to oversimplify the tensions among properties,” he wrote, and because partitions are rare, “there is little reason to forfeit C or A when the system is not partitioned” (Brewer). The line from that essay I find most useful is this one: “the essence of CAP takes place during a timeout.” Every time a system waits on another and gives up, it has made a CAP decision, whether or not anyone called it that.

Kleppmann adds the part that mattered most for us: “the CAP theorem says nothing about latency.” Daniel Abadi made the same point in 2012 with a longer acronym, PACELC. If there’s a partition, a system trades availability against consistency, else, when everything is running normally, it still trades latency against consistency (Abadi). Most of the time, that second trade-off is the one you’re actually living with.

The Feed Had Already Chosen

The NHL doesn’t publish documentation for its API, but its responses describe themselves. The feed today is served through Cloudflare, and a request for the live scores comes back with headers like these:

cache-control: no-transform, max-age=19, s-maxage=19
age: 11
cf-cache-status: HIT

That response is a cached copy, eleven seconds old, allowed to be up to nineteen, served from whichever edge location you happened to reach.

We read the NHL’s stats API as it was circa 2023, rather than its game-center data, because game center didn’t update during a game. The API has changed since, and its old responses are gone, but the shape of the problem survives. Here’s a trimmed goal from today’s score API, from a real game, Utah at Anaheim on October 16, 2024, which the Ducks won 5–4 in overtime:

{
  "period": 1,
  "periodDescriptor": { "number": 1, "periodType": "REG", "maxRegulationPeriods": 3 },
  "timeInPeriod": "04:40",
  "playerId": 8478873,
  "name": { "default": "T. Terry" },
  "teamAbbrev": "ANA",
  "assists": [
    { "playerId": 8476458, "name": { "default": "R. Strome" } },
    { "playerId": 8478366, "name": { "default": "F. Vatrano" } }
  ],
  "awayScore": 0,
  "homeScore": 1,
  "strength": "ev"
}

Look for the goal’s ID. There isn’t one. The NHL’s game-center endpoints give the same goal an eventId today, 157 in this game, but the goals in the feeds built for live scores are just a list of facts: a period, a time, a scorer, a team. That’s what we had to work with too. During a live game, any of those facts can come back from an older cache than the poll before it, or be corrected a few minutes later.

Goals Had No ID

That was part of what made this hard. Without an ID, the only way to know that the goal in this poll is the same goal as one in the last poll is to compare the facts, something like the game, the period, the time in the period and the team. Most of the time that works. But the facts are exactly what gets corrected. If the official scorer moves a goal from 4:40 to 4:38, or credits it to a different player, the goal you already announced no longer matches anything, and a new goal you’ve never seen appears in its place. To a naive notifier that’s a goal overturned and a new goal scored, two notifications for no event at all.

The settle window absorbed most of that, because corrections tend to land within it. The impossible-state checks below caught more. And it’s the reason deduplication downstream mattered so much. When identity is inferred rather than given, you have to assume you’ll sometimes infer it twice. Poll it twice a few seconds apart and the two answers can come from different caches of different ages. That’s a system that chose availability and low latency over consistency, and for a feed serving the whole internet, it’s the right choice. Every request gets a fast answer. Nobody gets a guarantee that it’s the latest one.

We couldn’t change that. We were a reader of someone else’s design, and no amount of polling would turn an eventually consistent feed into a linearizable one. The only decision left to us was the one Brewer described, how long to wait before treating what we saw as true.

Measuring the Window

So we measured it. Instead of guessing how long the feed took to settle, we watched how long updates kept changing after they first appeared. The answer was consistent. Almost all of them had settled within 45 seconds.

That number is the whole lesson. Eventual consistency is usually described as if “eventually” were unbounded, and in theory it is. In practice, for a particular system under a particular load, it has a shape, and you can measure it. Ours said that if a state change survived 45 seconds, it was almost certainly real.

We gave it more than double. Our working assumption was that most errors in the feed would be corrected within two minutes, and that everything in it was eventually consistent, so we set the window just under it. To be as close to real time as we could, we picked a minute and forty-five seconds, and relied on deduplication and idempotency downstream to absorb anything that slipped through a slightly shorter wait. The window also wasn’t the whole delay. Firebase added its own latency getting a notification out to phones, so every second we spent waiting on the feed was stacked on top of the time it took the push to arrive, and that was the other reason to shave the window as thin as we safely could. Notifications came from the goals in the Stats API, with the period alongside them. We kept the last state we’d committed, and on every poll we compared it with the new one. When they differed, we marked the change and started a clock. If that part of the feed changed again, its clock started over, because the 1:45 ran from the last time we’d seen it change, not from the first. Every game state change got its own clock, each goal and each period change separately, and the only question each clock answered was whether that change stayed. A flickering goal for one team couldn’t hold up a clean goal for the other. And when 1:45 passed with nothing new, we compared again. If the change still existed, the new state still differing from the committed one, we committed it, and the committed state went out through Azure Event Hub to everything that sent notifications. If the feed had settled back to what we already had, there was nothing to commit and nothing to send.

We polled the feed on a cron every 15 seconds, so every change was seen within 15 seconds of the feed showing it, and each poll ran the same check. Simplified, the loop looked something like this:

const SETTLE_MS = 105_000; // 1:45

// One entry per game state change we track: each goal we've ever seen, and the period.
type Key = string;   // e.g. "goal:1|04:40|ANA" or "period"
type Value = string; // e.g. "present", "gone", "P3", "OT", "FINAL"

const committed = new Map<Key, Value>();                          // what fans have been told
const pending = new Map<Key, { value: Value; since: number }>();  // changes waiting to prove they stayed

function onPoll(latest: Map<Key, Value>, now: number) {
  for (const [key, value] of latest) {
    if (committed.get(key) === value) {        // nothing new, or it changed back
      pending.delete(key);
      continue;
    }
    const seen = pending.get(key);
    if (!seen || seen.value !== value) {       // a new change, or it changed again: start its clock over
      pending.set(key, { value, since: now });
      continue;
    }
    if (now - seen.since >= SETTLE_MS && isPossible(latest)) { // it stayed for 1:45
      publish(key, value);                     // idempotent downstream, keyed by game and key
      committed.set(key, value);
      pending.delete(key);
    }
  }
}

A goal that vanished and came back inside the window never reached a fan. A goal that was really overturned inside the window never reached a fan either, which is the point. We couldn’t tell those two apart in the moment, so we stopped trying to, and let time tell them apart for us.

In PACELC terms, we chose consistency over latency, on purpose, by a measured amount. “Real time” became “a minute and forty-five seconds behind the feed, and right.”

The fair counterargument is obvious. If waiting makes the notifications more correct, why not wait five minutes and be even more sure? Because a notification is only worth sending while it’s still news. Five minutes after a goal, the fan watching the broadcast has seen the replay twice, the fan at the Honda Center has watched another faceoff, and everyone else has already heard it from a friend. A notification that arrives after all of that is accurate and useless. And the wait isn’t the whole delay, since Firebase’s delivery time stacks on top of it.

And waiting longer doesn’t actually make it more correct. A stale cache or a bad entry can show up at the five-minute mark just as easily as at the one-minute mark, so a longer wait can still end on a bad state change. It doesn’t help with the NHL’s false transitions either. When the feed cycles through overtime on its way to a regulation final, it does that whenever the game ends, not on any schedule a fixed wait could outlast. A longer clock just means you start waiting later and still have to ask whether the change stayed. What made a change trustworthy wasn’t how long we waited before looking. It was that the change stayed, through a window long enough to cover how long the feed takes to disagree with itself, and we’d measured that: almost everything settled within 45 seconds. Past that point, the length of the window doesn’t buy correctness. It only decides how real time the notification is. The bad states that slip through at any length are what the impossible-state checks and deduplication are for.

Here’s the same rule run against an illustrative ten minutes of a game, polled every 15 seconds the way we polled. Drag the settle time down to zero and you get a notifier that tells fans everything the feed says, the moment it says it. Drag it back up and watch what that costs.

Settle time 2:00

a poll that differs from the one before ita true notificationone a fan would later see undone

An illustrative game, not a recording. The polls include the failures we saw in the real feed: stale reads that briefly undo a goal, a goal overturned on review, and the period cycling through overtime before showing Final in a game that never went to overtime.

Impossible States

Waiting handled staleness. It didn’t handle the other thing we found, which was that sometimes the NHL’s data was simply bad. Not old, wrong. So alongside the window, we checked every state we took for things that couldn’t happen in a hockey game.

The examples below are the kind of rules we wrote, in the feed’s own terms, not a full list of them. A team’s score can go down, because goals get overturned, but only if a goal disappears from its list with it, so a score that dropped while every goal was still listed was a bad read, not a retraction. The period number in periodDescriptor never goes backward. The clock never runs backward within a period. A team’s score should equal the number of goals in the list with its teamAbbrev, and the last goal’s homeScore and awayScore should match the score. And a goal’s playerId has to belong to someone who’s actually playing for that team. A state that failed any of those checks wasn’t published, whether it had survived the window or not, and the next good state replaced it. Backward moves were the easiest case of all. If the period went backward, or the clock ran backward, we didn’t have to wait out any window to decide. We knew that state was wrong, so we tossed it the moment we saw it, and it never touched a clock or the committed state.

That turned out to be the more useful idea of the two. The window protects you from a system that’s eventually consistent. The invariants protect you from a system that’s eventually wrong, and they also catch the stale reads the window misses, because a stale copy of a game often contradicts itself in exactly these ways. Brewer’s advice about recovery says the same thing from the other side. The system “discovers the violation during recovery,” which only works if you’ve written down what a violation is.

Periods Were Hard

Periods were where the feed was at its worst, and the end of the game was worst of all. In games that ended in regulation, the feed would often cycle through overtime on its way to final. It would show the third period ending, then a fourth period with a periodType of OT, then the third again, then OT, before finally settling on the game being over, in a game that never went to overtime. A naive notifier would have told fans the Ducks were headed to overtime, twice, and then that they’d won in regulation. Similar cycles showed up at other transitions. The restarting clock is what saved us. Every flip counted as a new change and pushed the commit back, so we only committed once the state held still for the full 1:45. The cost was that these notifications were always the latest ones we sent, sometimes well past 1:45, because the feed took longest to agree with itself exactly when the most was changing at once: the clock running out, a goal at the buzzer, the game ending.

The other kind of churn was corrections. Points were sometimes entered wrong and fixed later, an assist credited to the wrong player, a goal reassigned after review. A correction that arrived inside the window simply became part of the state we committed. One that arrived after it was just another change, marked, timed and committed on its own, the same as any other.

In Order, Exactly Once

Settling the state was half the problem. The other half was getting each committed change to a fan’s phone in order and only once, through a queue that led to Firebase, which sent the push notifications. Consistency doesn’t end at the source. A correct goal delivered before the period it was scored in, or delivered twice, is just as wrong as a stale one.

Order came from how we published to Azure Event Hub. Each committed change went out with the game as its partition key, and Microsoft’s documentation is direct about what that buys you: “Specifying a partition key enables keeping related events together in the same partition and in the exact order in which they arrived” (Microsoft). Every event for one game sat in one ordered log, so the consumer sending to Firebase read a game’s changes first in, first out.

Exactly once is the part no queue gives you for free. Event Hub consumers track their place with checkpoints, and when a consumer restarts it resumes from its last checkpoint, which means anything it processed after that checkpoint and before the restart gets processed again. So every committed change carried an idempotency key, the game and the sequence number of the commit, and the consumer checked it before sending anything to Firebase. A goal that came through twice was recognized the second time and dropped. A retry was harmless, which is the only safe kind of retry.

How We Tested It

A rule like this is impossible to test against a live game. A game happens once, the interesting moments last seconds, and by the time you notice a bug the moment is gone. The obvious alternative is a simulator, a fake feed that serves stale reads and flapping periods on demand. Why not build one? Because the hard part of the problem is exactly the part you can’t fake cheaply. Reproducing how a globally cached feed actually disagrees with itself, which edges are stale, by how much and in what order, would have been a large engineering project of its own, and it would still only reproduce the failures we already knew to imagine. We didn’t have to imagine them. We knew when they happened. So we recorded them. The same 15-second cron that polled the feed also captured every raw API response for a game, exactly as the NHL served it, stale reads and bad data included. Afterward we could replay a whole game through our own pipeline, at whatever speed we wanted, and watch what it would have sent.

Replays turned the feed’s worst behavior into test cases. A period that cycled three times was a recording we could run again after every change to the rule. A goal that vanished for one poll and came back was a fixture. When we changed the settle time or added an impossible-state check, we replayed the games we had and compared what went out. Capture at the moments you know are broken, replay them as often as you like, and the real system’s real failures become a regression suite. That’s what let us build something resilient to a feed we couldn’t control. It also let us push toward near real time safely. Every time we tightened the settle time, the recordings told us what the tighter window would have gotten wrong, before a fan ever saw it.

Recordings only show you the feed, though, and the feed was the thing we didn’t trust. So for two or three games, we had someone in the building at the Honda Center, noting when things actually happened on the ice. Lining their notes up against the recordings is what showed us how far behind the feed really was. Real events were taking two to three minutes to settle into a state we could trust, which is the gap we were designing for. You can’t measure the lag of a source from inside the source.

What I’d Keep and Change

Brewer’s essay has an answer for what we didn’t build. After a partition heals, he wrote, the system “must create compensating operations” to correct mistakes it made while it was guessing. Banking, in his example, “depends not on consistency for correctness, but rather on auditing and compensation.” A notification system can work the same way. Send the goal quickly, and send a correction if it’s overturned. That’s the availability side of the trade, and it’s the right one for some products. It wasn’t ours, because a correction is still a fan who got told something false, and for a team’s own app that cost more than 1:45 did.

If I were building it again, I’d start the same way, by recording the source and measuring how long it takes to settle instead of guessing, with someone in the room for a few games to check it against reality, because that number turns an argument about consistency into a decision about latency. I’d still choose the wait for anything a fan would be embarrassed to have celebrated, and I’d write the impossible-state checks first, because they caught both kinds of bad data, the stale and the simply wrong. I’d treat compound moments like the end of a period as their own case from the start, rather than as a pile of single events. And I’d stop saying “real time” in planning meetings without a number attached, because every real-time system is some amount of time behind something.

Every feed is eventually consistent. Real time is just the number of seconds you decide to wait.