Realtime Agent Architecture

8 min read

V1
Playbook · Voice & Realtime Agents

Realtime agent architecture: the session's context window is a clock, and it runs out before your call does.

A voice agent's hardest constraint is not latency — it is that audio fills the model's context at a fixed rate, so a realtime session has a maximum length measured in minutes and set by arithmetic you can do in advance. Google documents about 25 tokens per second for streamed audio against a 32k window on most Live models, which is why the published session caps land where they do. Pick the speech boundary — full cascade, half-cascade, or native audio — only after you know which of the three clocks your call hits first.

Step 1The cascade: three models in a streaming pipeline.

The classic architecture is a cascade: a streaming speech-to-text model transcribes the caller, the transcript drives a normal text LLM agent loop, and a text-to-speech model voices the reply. It is the same architecture as a chat agent with a microphone on the front and a speaker on the back, and it still wins more production arguments than the newer options.

  • Transparent. The transcript and the text turn are both inspectable — you can log, eval and guardrail them with the exact tooling you already have for text agents.
  • Composable. Swap STT vendor, LLM or TTS voice independently. Best-of-breed at each stage, and a bad vendor is a config change rather than a rewrite. The speech stack prices each hop.
  • Lossy at the seams. Prosody, emotion, hesitation and overlap are flattened to text and gone. The model never heard the caller; it read a transcript of them, and whatever the words did not carry is unrecoverable downstream — the general form of that trade is in modalities and multimodal models.
  • Additive latency. Three hops, three queues. The budget arithmetic is in the latency budget; the short version is that a cascade spends its time on serialisation and a native model spends its on the model.

Step 2There are three places to put the speech boundary, not two.

The familiar framing is cascade versus a single speech-to-speech model that takes audio in and emits audio out. The vendors have since split the middle, and the middle option is the one most production stacks end up on.

# full cascade: three hops, three failure points, yours to swap
mic → STT → LLM (agent loop) → TTS → speaker

# half-cascade: provider-hosted, audio in / text reasoning / audio out
mic → realtime session [ audio-in → text core → TTS ] → speaker

# native audio: one model, audio in / audio out, no text on the path
mic → speech-to-speech model (loop inside) → speaker

The choice between the last two is not a preference, and the clearest statement of the trade comes from the provider. Google's own Live API guidance recommends half-cascade "for applications prioritizing tool execution reliability or needing text output", and native audio "for natural conversation with emotional awareness" — and notes that native-audio models must output audio, while half-cascade models support both modalities. Native audio gets the expressive features (tone matching the user's expression; the model choosing when to speak) that half-cascade lacks.

Read that as the vendor telling you that the expressive path is the less reliable one at calling your functions. If your agent's job is mostly to look things up and write things down, that is the whole decision; see voice tooling and state for what tool calls do to a turn.

Native does not mean transcript-free in practice. You still want a transcript for logging, eval and compliance — most realtime APIs emit one as a side channel. The difference is that it is observability, not the reasoning substrate. Do not put a guardrail on a transcript and assume it gates audio the model already spoke.

Step 3Audio fills the context window at a fixed rate, so the session cap is arithmetic.

This is the constraint that has no analogue in a text agent, and the one that decides whether your architecture survives a long call. A realtime session accumulates the conversation's audio in the model's context window, and it accumulates at a rate set by the medium rather than by the speaker. Google documents roughly 25 tokens per second for streamed audio and 258 for video, against a context window of 128k tokens for native-audio-output models and 32k for other Live models.

Divide one by the other and the documented session limits stop looking like policy:

32k window  ÷ 258 TPS (audio+video)  ≈ 124 s   → documented cap: ~2 min
32k window  ÷  25 TPS (audio in)     ≈ 1280 s  → documented cap: ~15 min
                                                  (the agent's own speech
                                                   consumes the window too)
128k window ÷  25 TPS (audio in)      ≈ 5120 s  → ~85 min of one-way talk

The two-minute audio-plus-video cap is almost exactly 32k divided by the video rate. The fifteen-minute audio cap is consistent with the same window once you account for both sides of the conversation landing in it. Treat the cap as a context budget rather than a timer, because that is what tells you what the remedies buy: context-window compression lifts the session cap because it reclaims window, not because it relaxes a rule.

Three consequences follow immediately. A call that can run long needs compression enabled from the start, not added after a truncation incident. A native-audio model buys you four times the window as well as the prosody, which is a second and rarely-cited reason to pick it. And an agent that streams video — screen share, camera — has a session measured in a couple of minutes, so video is a burst input you open and close, never a thing you leave running.

Step 4The transport expires before the session does.

Text agents move JSON over HTTP. Voice agents move continuous audio frames over a persistent connection for the entire call, and there are three shapes of that connection — WebRTC for browser and app clients, a WebSocket for server-to-server, and SIP for the phone network. OpenAI's Realtime API accepts all three, and its audio rates do not change with the transport, so the choice is an engineering one: WebRTC brings jitter buffering, echo cancellation and packet-loss concealment that you would otherwise write yourself, which is why a WebSocket is the wrong default for anything with a microphone on the far end.

SESSION  open WebRTC/WS/SIP ---------------------- hang up
         |  audio frames in (20ms)  audio frames out  |
         |  events: speech_started, transcript, ...    |
         |  function_call / function_result           |
         |  one conversation state, several connections |

The last line is the change from the naive picture. The connection and the session have different lifetimes: Google documents individual WebSocket connections timing out at around ten minutes while a compressed session can run far longer, and session resumption is disabled by default — you enable it in the setup message, keep the latest token from the resumption-update messages, and pass it as the handle when you reconnect. Ship without that and your first ten-minute call ends as a dropped conversation with no state to return to.

OpenAI's cap is a session cap rather than a connection cap: its documentation puts the maximum duration of a Realtime session at 60 minutes, raised from an earlier 30. Either way the architectural conclusion is the same — a call longer than the vendor's cap is a requirement to be designed for, with a handoff or a re-established session carrying summarised state, not an edge case to discover in production. Telephony adds its own hop: a SIP/PSTN gateway bridges the carrier to your media stream, usually as 8 kHz µ-law, and telephony and PSTN integration covers what that costs you in fidelity.

Step 5Where the agent loop lives.

With the clocks established, loop placement is a real decision rather than a taste. The loop — plan, call tool, observe, continue — has to run somewhere, and voice makes its placement load-bearing because every tool call happens inside a conversation that is still being billed by the second.

  • Cascade: the loop is yours, in your orchestrator, between STT and TTS. Full control of tools, memory and policy; you own the latency and the context accounting too.
  • Native, model-driven: the loop runs inside the realtime model, which decides when to call your tools via function calling over the session. Lowest latency, least control — and the configuration the provider itself flags as the weaker one for tool reliability.
  • Native, externally-driven: the model handles speech and turn-taking, but hands reasoning-heavy steps to an external text agent. This keeps audio quality and reclaims tool and policy control, at the cost of a round-trip you must hide behind a filler turn — see turn-taking and barge-in.

One asymmetry is easy to miss: in the two native shapes, the audio history occupying the context is not yours to prune. In a cascade, the text turns are yours — you can summarise, drop and re-order them with ordinary context-engineering tools, which means the session-length clock in STEP 3 is a clock you control. That, and not transcript visibility, is the strongest reason a long-running or heavily-tooled voice agent still ends up as a cascade.

Step 6The reference shape, and when not to build one.

Robust stacks converge on five load-bearing components regardless of which boundary you picked: a media server holding the call, a turn detector deciding who speaks, the model producing the reply, a tool runtime off the audio path, and a side channel for transcript, recording and async checks.

# the load-bearing components, not a framework
media_server   # holds WebRTC/SIP, jitter buffer, echo cancel
turn_detector  # VAD + semantic endpointing (see V3)
brain          # native S2S model OR cascade orchestrator
tool_runtime   # your functions, run off the audio path (V5)
side_channel   # transcript, eval, recording, async checks

Turn detection is worth one note here because it is the component most often left on its default. Voice activity detection is enabled by default on the major realtime APIs, and the newer semantic variants endpoint on what the caller has said rather than on how long they have been quiet — which is the difference between interrupting someone mid-sentence and waiting through a pause. Turn-taking and barge-in is the whole essay; voice failure modes is what each component looks like when it breaks.

And if the task tolerates a typed turn, voice buys you nothing and costs you a real-time deadline, an audio failure surface, a per-second bill that runs through silence, and worse observability. Text-first; voice only when the channel is the requirement. The per-second arithmetic is in the cost of a voice minute.

Decide in this order, not the usual one. First work out your maximum call length and divide your target model's context window by the audio token rate — if the answer is shorter than your calls, you need compression or a native-audio window before anything else matters. Second, enable session resumption and test a reconnect mid-call, because the transport will expire before the session. Only then choose the speech boundary: cascade when you need transcript-level control, eval parity with text agents, heavy tooling, or a context window you can prune yourself; half-cascade when you want the provider's realtime loop and your functions to be called reliably; native audio when conversational quality is the product and the tool surface is small.