Engineering an AI Transport Layer

Most engineers who haven’t worked on real-time systems would reach for WebSockets to build a chat agent. It’s the obvious choice. The model streams tokens, the user sends messages, and WebSockets are the tool for two-way, real-time traffic.

It’s usually the wrong one. An AI transport layer has to do three things WebSockets were never designed for: stream partial results with bounded latency, multiplex many concurrent generations over one connection, and recover cleanly when either side drops mid-token. Most teams find this out after they’ve already built the first version on raw WebSockets.

What WebSockets Actually Cost

A WebSocket gives you a pipe of messages and nothing else. Every frame carries a header of 2 to 14 bytes, and a message can be split across frames that have to arrive in order and be reassembled before anything can read them (RFC 6455). Your library handles that part. What it doesn’t give you is anything a chat agent needs: no event types, no message IDs, no way to say “I last saw token 412, start from there.”

So every team builds an envelope, and the envelope is almost always JSON:

{"type":"token","session":"s1","generation":"g7","seq":412,"text":" the"}

That’s about seventy bytes of bookkeeping to carry four bytes of token. And JSON only works as one atomic unit. It can’t be parsed until it’s complete, so anything that streams in pieces, like the arguments to a tool call, has to be buffered until the closing brace arrives. What looks like a little added latency at low concurrency becomes a memory and bookkeeping problem at scale. Every open session carries partial-message state, and allocating, retaining and eventually reclaiming that state starts to show up in both your tail latency and your bill.

Then there’s the protocol. If you want to speak something structured over the socket, ACP or MCP, say, you’re writing the request IDs, the correlation and the error handling yourself unless a library does it for you. And the connection itself is yours to manage: heartbeats, reconnects, backoff, auth on upgrade, and what happens to the generation that was halfway done when the socket died. None of it is hard on its own. All of it together is easy to misconfigure, and it takes real QA discipline to get right.

To be fair to WebSockets, they’re the right tool when the client streams continuously too: voice, collaborative cursors, multiplayer anything. A chat agent isn’t that. The user sends a message every so often, and the server streams back a lot.

POST and SSE

The more reliable option is the one the frontier labs use. Both Anthropic and OpenAI stream their APIs the same way: the request is a regular HTTP POST, and the response is a stream of server-sent events. Mostly text, occasionally JSON.

SSE solves the three problems almost for free.

Streaming. An SSE event is a few plain-text lines: an optional id:, an optional event: name, and data:. The event name is your envelope. A token is event: token followed by the token text, with no JSON around it. The only JSON you ever buffer is the JSON that actually is JSON. Anthropic’s own stream does exactly this, sending tool arguments as partial_json deltas inside a typed event.

Multiplexing. Over HTTP/2, every stream is its own HTTP/2 stream on one shared connection, and the browser and server negotiate how many can run at once, 100 by default. The multiplexing you’d build into your WebSocket envelope is already in the protocol. One warning: over HTTP/1.1, browsers cap you at six SSE connections per domain across all tabs, a limit Chrome and Firefox have marked “won’t fix.” Serve it over HTTP/2.

Recovery. If the connection drops, the browser reconnects by itself and sends the ID of the last event it saw in a Last-Event-ID header (WHATWG). The resume protocol you’d otherwise design is already written. All the server has to do is be able to answer “everything after this ID,” and that turns out to be the most important decision in the whole design.

The shape I use is two endpoints. The client POSTs a message with its session token, and it holds one long-lived GET open for the session’s stream. The POST starts a generation and returns right away. The GET is what carries the tokens, and it’s the one that reconnects.

Managing Connections Across Nodes

Once you have more than one server, you need to think about two or three things, depending on the setup:

1. Who owns the connection state. A database, a cache like Redis, or the node’s own memory.
2. How many connections you plan to have open at once.
3. Your ingress and load-balancing situation.

For a simple setup, keeping connections in memory in Go looks something like this:

// Connections holds every active session on this node.
type Connections struct {
    mu       sync.RWMutex
    sessions map[string]*Session // session ID -> session
}

// Session is the per-connection state.
type Session struct {
    tx          chan Event // buffered, e.g. make(chan Event, 256)
    userID      *string    // nil if not authenticated
    connectedAt time.Time
    lastSeen    time.Time
}

For most SSE setups the sessions map covers 90% of real use. A string key is fine. People sometimes hang more off the session: the IP or user agent, a flag to force-close it, per-session topics if clients subscribe to different events, rate-limit counters. Add those when you need them, not before.

The catch is the word memory. The stream lives on whichever node accepted the connection, so every request for that session has to land on the same node. Behind a load balancer that means sticky sessions. If you haven’t used them, the load balancer sets a cookie on the first response, and on every request after that it reads the cookie and sends you back to the same server.

Sticky sessions work until they don’t. When a node restarts during a deploy, every stream on it dies, and the generations die with them. The reconnect lands on a different node that has never heard of the session. And when the client’s network blips mid-token, Last-Event-ID doesn’t help, because the only copy of those tokens was in a channel on a node the client may never reach again.

POST → Tokens → Redis

The fix is to stop letting the node that’s generating own the stream. The generation writes its tokens to a Redis stream, and whichever node holds the client’s connection reads them back out.

client ──POST──▶ node A ──XADD──▶ Redis stream ◀──XREAD── node B ──SSE──▶ client

Redis Streams fit SSE almost exactly. Every entry gets a monotonically increasing ID, and XREAD returns every entry after an ID you give it, blocking until one arrives. That ID is your SSE id:. When the browser reconnects with Last-Event-ID, you pass it straight to XREAD, and the stream picks up from the token after the last one the user saw. No sticky sessions, no lost tokens, and a deploy is just a reconnect.

The POST side starts the generation and gets out of the way:

// POST /sessions/{id}/messages
func (s *Server) SendMessage(w http.ResponseWriter, r *http.Request) {
    var msg struct {
        Text string `json:"text"`
    }
    if err := json.NewDecoder(r.Body).Decode(&msg); err != nil {
        http.Error(w, "bad request", http.StatusBadRequest)
        return
    }

    // The generation outlives the request. If the client drops,
    // the tokens still land in Redis for whoever reconnects.
    ctx, cancel := context.WithTimeout(context.WithoutCancel(r.Context()), 5*time.Minute)
    go func() {
        defer cancel()
        s.generate(ctx, r.PathValue("id"), msg.Text)
    }()
    w.WriteHeader(http.StatusAccepted)
}

func (s *Server) generate(ctx context.Context, sessionID, prompt string) {
    key := "chat:" + sessionID
    emit := func(event, data string) {
        s.rdb.XAdd(ctx, &redis.XAddArgs{
            Stream: key,
            MaxLen: 10_000,
            Approx: true,
            Values: map[string]any{"event": event, "data": data},
        })
    }

    for token, err := range s.model.Stream(ctx, prompt) {
        if err != nil {
            emit("fail", err.Error())
            return
        }
        emit("token", token)
    }
    emit("done", "")
    s.rdb.Expire(ctx, key, 10*time.Minute) // long enough to reconnect; the finished answer lives in your database
}

The line that matters is context.WithoutCancel. Tie the generation to the request’s context and a dropped connection cancels the model call halfway through. Detach it and the generation finishes no matter what the client does.

The GET side reads from wherever the client left off:

// GET /sessions/{id}/stream
func (s *Server) Stream(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    key := "chat:" + r.PathValue("id")

    // The browser sends this on its own when it reconnects.
    last := r.Header.Get("Last-Event-ID")
    if last == "" {
        // A fresh connection starts at the newest entry. Not "$": that's
        // re-evaluated on every read and drops anything written in between.
        last = "0-0"
        if top, _ := s.rdb.XRevRangeN(ctx, key, "+", "-", 1).Result(); len(top) > 0 {
            last = top[0].ID
        }
    }

    w.Header().Set("Content-Type", "text/event-stream")
    w.Header().Set("Cache-Control", "no-cache")
    rc := http.NewResponseController(w)
    rc.SetWriteDeadline(time.Time{}) // a stream outlives the server's WriteTimeout
    // Spread out reconnects, so a deploy doesn't bring every client back in the same second.
    fmt.Fprintf(w, "retry: %d\n\n", 1000+rand.IntN(4000))

    for {
        res, err := s.rdb.XRead(ctx, &redis.XReadArgs{
            Streams: []string{key, last},
            Count:   100,
            Block:   15 * time.Second,
        }).Result()
        if errors.Is(err, redis.Nil) {
            fmt.Fprint(w, ": ping\n\n") // a comment line keeps the load balancer from idling us out
            rc.Flush()
            continue
        }
        if err != nil {
            return // the client left or Redis is down; either way the browser reconnects
        }
        for _, m := range res[0].Messages {
            writeEvent(w, m.ID, m.Values["event"].(string), m.Values["data"].(string))
            last = m.ID
        }
        rc.Flush()
    }
}

var lineEnds = strings.NewReplacer("\r\n", "\n", "\r", "\n")

func writeEvent(w io.Writer, id, event, data string) {
    fmt.Fprintf(w, "id: %s\nevent: %s\n", id, event)
    // A raw newline in a token would end the event early, so each line
    // gets its own data: field. The browser joins them back with "\n".
    for _, line := range strings.Split(lineEnds.Replace(data), "\n") {
        fmt.Fprintf(w, "data: %s\n", line)
    }
    fmt.Fprint(w, "\n")
}

Two details in there bite people. Models emit newlines as tokens all the time, and in SSE a blank line ends an event, so a raw "\n\n" in the data cuts the message in half. Splitting each line into its own data: field is the spec’s own answer. The other is the 15-second ping. An AWS Application Load Balancer closes a connection after 60 seconds with no data by default, and it doesn’t count HTTP/2 PING frames as data. An SSE comment line does count.

On the client, the GET is a plain EventSource, which handles the reconnect and the Last-Event-ID for you:

const stream = new EventSource(`/sessions/${id}/stream`, { withCredentials: true });
stream.addEventListener("token", (e) => append(e.data));
stream.addEventListener("done", () => finish());
stream.addEventListener("fail", (e) => showError(e.data));

async function send(text: string) {
  await fetch(`/sessions/${id}/messages`, {
    method: "POST",
    credentials: "include",
    headers: { "content-type": "application/json" },
    body: JSON.stringify({ text }),
  });
}

Open the stream before the first POST, and don’t name a server event error. EventSource fires its own error event when the connection drops, and you don’t want to read a dropped connection as a failed generation. EventSource also takes a URL and nothing else: no method and no custom headers. That’s why the session rides in a cookie, and why the labs’ own clients use fetch and parse the stream themselves. For the GET half of this design, EventSource is all you need.

So where does the Connections map go? It stays, but it stops being the source of truth. It becomes the node’s local fan-out, the list of who’s listening on this machine. Redis is what remembers.

Using GraphQL Subscriptions

If you’re already on GraphQL, a subscription gives you a strong contract for the different states of a chat, because GraphQL can send each message as its own type:

union ChatEvent = TokenMessage | DoneMessage | ErrorMessage

type TokenMessage { token: String! }
type DoneMessage { stopReason: String! }
type ErrorMessage { error: String! }

type Subscription {
  messages(session: ID!): ChatEvent!
}

subscription {
  messages(session: "s1") {
    __typename
    ... on TokenMessage { token }
    ... on DoneMessage { stopReason }
    ... on ErrorMessage { error }
  }
}

The client gets generated types for every state, so a missing case is a type error, not a production bug. Subscriptions don’t have to mean WebSockets either. gqlgen ships an SSE transport that speaks the graphql-sse protocol, so it’s one line, srv.AddTransport(transport.SSE{}), and you keep everything above. The resolver reads from the same Redis stream.

The trade-off is the envelope. Every token is now a small JSON object again. For a BI tool or an internal agent, the contract is worth it. For a consumer chat streaming to millions, I’d keep the token path plain and put the typed contract on everything around it.

Figuring Out How Many Nodes

Connections are the wrong place to start, because a chat app has two different numbers hiding in it, and they cost different things.

The first is open streams. Every open tab holds one GET, whether it’s generating or not. Call that $C$. The second is generations in flight, $G$, and that comes from Little’s Law: the average number of things in a system equals the rate they arrive times how long each one stays.

Say you peak at 30 messages a second and an answer takes 20 seconds to stream. That’s about what our BI agent took.

$C$ is just how many people have the app open. Say 10,000. I’ll also assume a model streams 50 tokens a second. Swap in your own numbers; the method is the point. Now walk every resource and see which one breaks first.

Memory on the Node

An idle stream in Go is a goroutine, which starts with a 2 KB stack, plus the server’s buffers, the TLS state and the kernel’s socket buffers. Budget a generous 64 KB per open stream, and measure yours.

One node holds that. Memory doesn’t set the count.

Bandwidth

A token event on the wire is about 45 bytes: the id: line, event: token, data: with the token, and the blank line that ends it.

Not it either.

Redis Connections

This is the one that bites. In the handler above, every open stream holds a blocking XREAD, and a blocked XREAD holds a Redis connection the whole time it waits. That scales with $C$, not $G$.

You’d hit the ceiling at exactly the traffic in this example, and adding nodes wouldn’t move it. You’d hit go-redis first, though. Its pool defaults to 10 connections per CPU, so a 4-core node has 40, and stream 41 waits in line.

The fix is to stop blocking per stream. Run one reader per node that does a single XREAD across every session that node is serving, and fans the entries out through the Connections map. Redis connections drop from $C$ to the number of nodes. That’s the job the map was waiting for. (On Redis Cluster, one XREAD can only span keys in the same hash slot, so it’s one reader per node per shard.)

Redis Memory

Every token is a stream entry, and every answer is 20 seconds × 50 tokens a second = 1,000 entries. How many you keep is Little’s Law again: entries held equal the rate you write them times how long each one lives. Say you average 10 answers a second over a day, and an entry costs 50 bytes (check yours with MEMORY USAGE).

The stream only needs to live as long as a reconnect might need it. Write the finished answer to your database and let Redis forget the tokens. That’s why the code above expires the key after 10 minutes, not a day.

File Descriptors

Every stream is a socket, and every socket is a file descriptor.

Go has raised its soft limit to the hard limit on its own since Go 1.19, so the number that matters is the hard limit your container or systemd unit sets. Check it before the load test, not after.

Deploys

This is where the node count actually comes from. When a node restarts, its streams reconnect to the ones still up, so with $N$ nodes each survivor goes from $C/N$ to $C/(N-1)$.

All of those land within the browser’s reconnect delay, which the spec leaves at “a few seconds.” Each one costs a catch-up read. Size every node for $C/(N-1)$, not $C/N$, and send a retry: field with some jitter so a restart doesn’t turn into everyone reconnecting in the same second.

Idle Timeouts

Every proxy between you and the browser has one, and the ping has to beat the shortest. Behind an ALB at its 60-second default, a 15-second ping gets four chances.

Your Model Provider

Put that next to your provider’s output-tokens-per-minute limit. It usually caps you long before your own servers do.

Add it up and memory says one node, bandwidth says one node, and deploys say at least three. Redis says the design has to change before the count matters at all.

Takeaways

If I were starting a chat agent today, I’d skip the WebSocket. I’d POST messages, stream back over SSE on HTTP/2, and let the protocol do the multiplexing and the resume that I’d otherwise write myself.

I’d keep the stream out of the node’s memory. Tokens go to Redis, and the SSE event ID is the Redis entry ID, so a dropped connection or a deploy is just a reconnect. I’d detach the generation from the request so it finishes no matter what the client does.

And I’d do the math before the load test. Count generations with Little’s Law, not sockets. Keep Redis connections per node, not per stream. Keep tokens only as long as a reconnect needs them. And size every node to absorb the one that’s restarting.

Sources

1. IETF, RFC 6455: The WebSocket Protocol, Base Framing
2. WHATWG, Server-sent events
3. MDN, Using server-sent events
4. Anthropic, Streaming messages
5. OpenAI, Streaming API responses
6. Redis, XREAD
7. gqlgen, Subscriptions
8. AWS, Application Load Balancer connection idle timeout
9. Go 1.4 release notes, goroutine stack size
10. Go 1.19 release notes, open file limit
11. Redis, Client handling and maxclients