Streaming & partial output.
Streaming does not make a model faster — it makes the wait legible, and on an agent that thinks for ninety seconds that is the difference between a product and a hung page. The cost is that you have committed to text you cannot take back: once a token is on the user's screen you cannot validate it, reorder it, or quietly retry, so every guardrail you own has to move to the front of the stream or accept that it now runs too late.
Two numbers, not one.
"Latency" hides the only distinction that matters for perceived speed:
- Time to first token (TTFT) — how long the user stares at nothing. Dominated by prompt processing, so it grows with input length and collapses when a prompt cache hits.
- Inter-token latency — the drip rate afterwards, roughly constant per model and hardware.
Total time is TTFT plus rate times length, and streaming changes none of it. What it changes is that the user starts reading at TTFT instead of at the end. A response that takes 20 seconds to generate but starts in 400ms feels responsive; the same response delivered whole at second 20 feels broken. This is also why the two levers are different jobs: shrink TTFT by caching and trimming input, shrink total time by generating fewer tokens.
Reading speed is roughly 5–8 tokens per second. Most hosted models emit faster than that, which means past a certain rate more speed buys nothing a user can perceive — while TTFT is felt at every single turn. Optimise the number people actually experience.
What an agent streams is not just prose.
Chat streaming is one text field. An agent's stream is a sequence of typed events, and treating it as a string is the mistake that makes agent UIs hard to build:
- Text deltas — the visible answer.
- Reasoning deltas — on reasoning models, thinking that arrives before any answer and may be summarised rather than verbatim. TTFT for the answer can be tens of seconds even when the stream opened instantly.
- Tool-call deltas — arguments arriving as a partial JSON string. You cannot act on them until the call is complete; there is no such thing as half a tool invocation.
- Lifecycle events — message start/stop, content-block boundaries, stop reason, and usage. The stop reason is the field that tells you whether you hit a natural end or a token ceiling.
The practical rule: render text deltas immediately, buffer tool-call deltas to completion, and surface tool activity as its own UI affordance ("searching…", "running tests…") rather than as text. A user watching an agent work is reassured by seeing what it is doing; they do not need the raw arguments.
The things streaming quietly breaks.
Every one of these is a real production failure, and they share a cause — you published before you finished:
- Output guardrails run too late. A moderation or PII check on the completed response cannot unsay what is already rendered. Either scan incrementally on a buffer and accept added TTFT, or accept that your output filter is now advisory. Input-side controls and guardrails do not have this problem, which is one more reason to weight them.
- Retries become visible. Non-streamed, a failed call is retried and nobody knows. Mid-stream, you have already shown half an answer, and the retry either duplicates or contradicts it. Decide up front: replace the block, or fail forward with a visible error.
- Partial JSON is not JSON. Structured output streamed to a parser is invalid for its entire life until the last token. Use a streaming-tolerant parser, or do not stream structured results at all.
- Disconnects are silent. A closed browser tab does not stop server-side generation; you keep paying for tokens nobody will read. Detect the dropped connection and cancel.
- Truncation looks like completion. A stream that ends because it hit the max-token ceiling looks exactly like one that finished, unless you check the stop reason. Always check it.
Transport and the shape it forces.
Server-Sent Events is the default for good reasons: it is one-directional (server pushes, which is exactly what generation is), it survives corporate proxies and HTTP/1.1 infrastructure, and it reconnects at the protocol level. WebSockets buy bidirectionality you rarely need for text — reach for them when audio is involved, where realtime models genuinely need a duplex channel.
- Buffering intermediaries defeat it. A proxy, CDN, or serverless platform that buffers the response body turns your stream back into a single delivery, and it will look fine in local testing. Verify against production infrastructure, not localhost.
- Long-running generation outlives request timeouts. Many serverless defaults sit below the time an agent takes to think. This is the most common reason a streaming agent works in development and 504s in production.
- Resumability is a design decision, not a feature you get. If a user should be able to close a laptop and come back, generation has to be a job with durable state, not a request holding a socket.
Stream text to humans; do not stream to programs. If the consumer is another service, a queue, or an evaluation harness, take the completed response — you gain validation, clean retries, and far simpler code, and you lose nothing, because no machine cares how quickly the first token arrived. Streaming is a user-experience mechanism that happens to run over your API.
Related: cost, quality & latency for the trade-off this sits inside, agent observability for what to record when the response arrived in pieces, and the agent loop for where tool events fit.