Schedule, Don’t Throttle

At AbbVie, a lot of things sent email. Provider onboarding, rewards notifications, booking confirmations, password resets, each from a different service, all through one SendGrid account, and many of them through Lambdas reading from SQS. That’s a good architecture right up until it meets a rate limit, because the two halves of it want opposite things. The queue and the Lambdas are built to drain as fast as possible. The email provider is built to tell you to slow down.

The usual fix is to throttle the workers. We did something else. We scheduled the sends before they ever reached a worker, so by the time a message got to one, it was already allowed to go.

Why the Workers Can’t Fix It

Lambda scales SQS consumers aggressively. AWS’s documentation says it starts with five concurrent invocations, adds “up to 300 more concurrent invokes per minute,” and that a single event source mapping can process up to 1,250 at once (AWS). A burst of a few thousand emails turns into hundreds of Lambdas sending at the same moment.

There are knobs, and none of them answer the real question. Since 2023, an SQS trigger can take a maximum concurrency, a number “between 2 and 1,000” that caps how many instances the queue can invoke, and once it’s reached Lambda “stops reading more messages from the queue” (AWS). That’s better than the older approach of reserved concurrency, which throttles invocations after messages are already received, so they bounce back to the queue and can end up in the dead-letter queue for no reason. But a concurrency cap limits how many workers run, not how many emails go out per second, and it only covers one queue. Three services with three queues and three caps can still add up to more than the account allows. A rate limit belongs to the account, so the thing enforcing it has to see the whole account.

The Design: Reserve a Slot, Then Wait for It

So we moved the decision out of the workers and into one place every sender shared. A central Redis held the org-wide budget, the number of sends per second the account could take. Anything that wanted to send an email didn’t ask whether it could send right now. It asked Redis for the next free slot, and Redis answered with a time. If the current second was full, the reservation rolled forward to the next second with room.

The message then went onto SQS with a delay that made it invisible until its slot. SQS calls these message timers. A timer gives a message “an initial invisibility period,” anywhere from zero up to 15 minutes (AWS). The queue became a schedule. A burst of ten thousand emails didn’t turn into errors. It turned into ten thousand delivery times spread across the next few minutes.

Here’s one email’s timeline. The burst is already ahead of it, so the next free second is twelve seconds out:

The reservation has to be atomic, because two services asking at the same moment must never get the same slot. Redis does this naturally with a short Lua script, which runs without any other command interleaving. The version worth copying keeps a cursor, the next second with room and how many sends it already holds, and moves it forward one email at a time, so every reservation costs the same no matter how big the burst. Here’s the shape of it, simplified, with the clock read from Redis itself so every service schedules against the same time:

-- KEYS[1]: the cursor, a hash of { second, used }
-- ARGV[1]: sends allowed per second (the budget, h·L)
local now    = tonumber(redis.call('TIME')[1])         -- one clock for every service
local limit  = tonumber(ARGV[1])
local second = tonumber(redis.call('HGET', KEYS[1], 'second') or now)
local used   = tonumber(redis.call('HGET', KEYS[1], 'used') or 0)

if second < now then second, used = now, 0 end          -- the cursor fell behind: start from now
if used >= limit then second, used = second + 1, 0 end  -- this second is full: move to the next

redis.call('HSET', KEYS[1], 'second', second, 'used', used + 1)
return second - now                                     -- the delay to put on the SQS message

The caller takes the return value as the message’s DelaySeconds, and anything further out than 15 minutes goes on with the maximum and is re-delayed when it surfaces. On Redis older than version 5, a script that writes after reading TIME has to call redis.replicate_commands() first. It’s the same idea as the per-second counter in Redis’s own rate-limiter pattern (Redis), turned from a yes-or-no check into a reservation for the next second with room.

The Math

The schedule is simple enough to reason about on paper, which is most of why I trust it. Call the provider’s real limit $L$ sends per second, and budget a fraction $h$ of it, the headroom, so the scheduler hands out $hL$ slots a second. If a burst of $N$ emails arrives at time $t_0$, the $k$-th reservation lands in the second

and is enqueued with a delay of $s_k - t_0$. The whole burst drains in

and because a message timer can’t exceed 900 seconds, the largest burst one hop of the queue can absorb is

The workers need surprisingly little concurrency. By Little’s law, the number of sends in flight is the arrival rate times how long each one takes, so with a send latency of $W$ seconds,

Put illustrative numbers on it. With a binding limit of $L = 100$ sends a second and 80 percent headroom, the scheduler hands out 80 slots a second. A burst of 10,000 emails drains in 125 seconds, the queue can absorb bursts up to 72,000 before messages need a second hop, and with sends taking a quarter of a second, about 20 workers are busy at any moment, far below the hundreds Lambda would otherwise have spun up.

The Dumb Worker

The worker was the best part, because it was dumb. When a message became visible, its slot had arrived, so the worker sent it. No sleeping, no counters, no coordinating with other workers. A hundred of them could run at once without knowing about each other, because the decisions had all been made upstream, once, in the one place that could see everything.

That’s the general principle, and it’s why I’d build it this way again. Rate limiting inside workers is hard because workers are many and blind. Scheduling ahead of them is easy because the scheduler is one and sees the whole account.

What Makes It Safe

The design is right, and it’s only safe if a few things are true.

It has to be the only door. A scheduler can only respect a limit it controls, which leaves two ways to build it. Either one caller owns the account, a single sending service everything else goes through, or every caller uses the same distributed Redis to reserve its slot. We did the second, so services kept their own queues and workers but shared one budget. Either way, a service that called SendGrid without a reservation would spend budget the scheduler didn’t know about, and the first sign would be a 429 nobody planned for. If one appears, something is going around the scheduler.

It needs a standard queue. Message timers only exist on standard queues, and AWS is explicit that “FIFO queues don’t support timers on individual messages.”

It needs headroom. A message timer is a minimum, not an appointment. Messages scheduled for different seconds can surface a little late and land together, especially while Lambda is still scaling up. Budgeting below the provider’s real limit, something like 70 to 80 percent of it, absorbs that bunching.

It needs one clock. If every service computed its delay from its own clock, skew between machines would shift the schedule. Reading the time from Redis inside the same atomic script, as above, means every reservation is made against one clock.

The worker is dumb, not blind. Scheduling prevents the expected failures, not every failure. The worker still needs to handle a 429 or a provider error by putting the message back with a delay, send repeated failures to a dead-letter queue, and protect against duplicates. Standard SQS queues deliver at least once, so the same message can arrive twice, and an email sent twice is a bug a customer sees. A send ID checked before sending, stored in the same Redis, closes that gap. And a retry has to go back through the scheduler. A message that failed and goes straight back on the queue spends budget nobody reserved, so it asks Redis for a new slot like any other email.

Here’s what that looks like in the worker, simplified. It’s still dumb. It sends, and it only has three outcomes to handle:

import type { SQSHandler } from 'aws-lambda';

export const handler: SQSHandler = async (event) => {
  const failures: { itemIdentifier: string }[] = [];
  for (const record of event.Records) {
    const email: Email = JSON.parse(record.body);
    if (await alreadySent(email.sendId)) continue;              // at-least-once delivery: skip duplicates

    const res = await sendgrid.send(email);
    if (res.ok) { await markSent(email.sendId); continue; }      // the normal case

    if ((res.status === 429 || res.status >= 500) && email.attempts < 5) {
      const delay = await reserveSlot();                          // a retry goes back through the scheduler
      await requeue({ ...email, attempts: email.attempts + 1 }, Math.min(delay, 900));
      continue;                                                   // this copy is done; the new one has a slot
    }
    failures.push({ itemIdentifier: record.messageId });          // anything else: let SQS retry, then dead-letter
  }
  return { batchItemFailures: failures };
};

alreadySent and markSent are a key per send ID in the same Redis with an expiry of a day or so, which is enough to catch SQS delivering a message twice. A 429 or a server error doesn’t retry in place, because retrying immediately is exactly the burst the scheduler exists to prevent. It reserves a new slot and goes back on the queue with that delay, and after a few attempts it stops, so a message that can never be sent ends up in the dead-letter queue instead of circling forever. Everything else is reported back through batchItemFailures, so only that one email is retried, not the whole batch. AWS’s own guidance for SQS-triggered Lambdas also applies: a visibility timeout of “at least six times” the function timeout, a maxReceiveCount of “at least 5” before messages move to the dead-letter queue, and partial batch responses with ReportBatchItemFailures, so one failed email doesn’t retry the whole batch (AWS).

It needs a plan for Redis being down. The scheduler is now a single point of failure for all email. Either hold mail in the queue until it comes back, or fall back to sending with a maximum-concurrency cap on the trigger as a blunt backstop. Either is fine if it’s decided in advance.

And it has a horizon. Message timers stop at 15 minutes, so a backlog bigger than that needs messages re-delayed when they surface, or a scheduler built for longer delays, which AWS points to EventBridge Scheduler for.

Which Limit, Exactly

One more thing worth being precise about, because it changes the budget. SendGrid documents its Mail Send endpoint generously, at “up to 10,000 requests per second,” with up to 1,000 recipients in a single request (Twilio SendGrid). That’s rarely the limit a team actually hits. Its other API endpoints have their own, much lower, limits per refresh period, after which further requests return a 429 until the period resets (Twilio SendGrid). And a new dedicated IP is warmed up gradually, with SendGrid capping how much it sends each hour, starting at 20 emails an hour on the first day (Twilio SendGrid). The scheduler doesn’t care which limit it’s respecting. The budget in Redis should be whichever one is actually binding, with the headroom on top.

Takeaways

If I were building outbound email for an organization again, I’d schedule rather than throttle. I’d put one shared budget in Redis, make reservations atomic and timed by Redis’s own clock, and let SQS message timers hold each message until its slot. I’d keep the worker dumb, and make it the only path to the account. And I’d still give it the basics a dumb worker needs, headroom under the real limit, retry with delay on a 429, idempotent sends and a dead-letter queue, because scheduling handles the traffic you expect, and those handle the rest.

Throttling asks every worker to be careful. Scheduling lets them all be simple.

Sources

1. AWS, “Configuring scaling behavior for SQS event source mappings,” Lambda documentation
2. AWS Compute Blog, “Introducing maximum concurrency of AWS Lambda functions when using Amazon SQS as an event source,” January 2023
3. AWS, “Amazon SQS message timers”
4. AWS, “Configuring an SQS event source mapping,” Lambda documentation
5. Redis, “INCR,” rate limiter pattern
6. Twilio SendGrid, “v3 Mail Send FAQ”
7. Twilio SendGrid, “Rate Limits”
8. Twilio SendGrid, “Warming Up an IP Address”