Voice & realtime agents.
A voice agent that hesitates half a second too long stops feeling like a conversation and starts feeling broken. Everything about building one bends around a single constraint — latency — and a single architectural fork: transcribe speech to text and back, or let one model hear and speak directly. This entry gives you both, plus the real-time control layer (turn-taking, interruptions) that is the part beginners always underestimate.
What a voice agent is — and why latency is the whole game.
A voice / realtime agent is an agent you talk to in spoken audio and that talks back in near real time: a natural back-and-forth where you can interrupt it, it can pause and think, and it can call tools mid-conversation to look something up or take an action. It is the same agent loop you have seen elsewhere, wrapped in a spoken interface and running against a stopwatch.
And the stopwatch is the point. In human conversation the gap between one person finishing and the next starting is only roughly 200–300 ms. A common target for a natural-feeling voice agent is to keep end-to-end voice-to-voice latency roughly sub-second — the time from when you stop talking to when you hear the first audio back. Miss that budget and the illusion of conversation collapses, no matter how smart the underlying model is. Latency is not one concern among many here; it is the defining constraint that shapes every other decision. (For the general framing of that trade, see cost, quality & latency.)
The common beginner mistake: "a voice agent is just speech recognition glued onto a chatbot." Recognition and synthesis are the easy, solved-ish parts. The hard part is the real-time conversational control — deciding when you have finished speaking, handling you talking over it, and keeping the whole loop under a tight latency budget. That is what the rest of this page is about.
The central choice: cascade vs speech-to-speech.
There are two ways to build the pipeline, and picking between them is the first real decision.
- Cascade (pipeline): STT → LLM → TTS. Speech is transcribed to text by a speech-to-text (STT) model, a normal text LLM reasons over that text (and calls tools), and the reply is synthesized back to audio by a text-to-speech (TTS) model. Three swappable stages.
- Speech-to-speech (realtime multimodal): one model ingests audio and emits audio directly, skipping the intermediate text bottleneck entirely.
Latency is exactly why both exist. In a cascade, total latency is the sum of every stage — STT + LLM + TTS + the orchestration between them — which structurally puts it at a disadvantage. Speech-to-speech collapses those stages into one pass, which is why it can be faster and why it exists at all.
But faster is not simply better, and this is the tradeoff to internalize:
- Speech-to-speech wins on latency and on prosody — because the model hears tone and emotion directly and produces tone directly, it sounds more natural. What you lose is observability and control: there is no clean transcript to grade, you cannot independently swap in a better STT or TTS, and tool-calling and structured output are weaker because they must be encoded through audio.
- Cascade wins on control, observability, compliance, and component choice — you can inspect the transcript, pick a specialist vendor for each stage, debug per stage, and use any LLM you like with reliable tool-calling. The cost is added latency and the loss of paralinguistic nuance (the tone gets flattened into text and rebuilt).
It's a tradeoff, not an upgrade. "Speech-to-speech is the advanced architecture and cascade is legacy" is a myth. In production, cascade is often the preferred choice — precisely because you can read the transcript, swap components, satisfy compliance, and rely on solid tool-calling. Choose by what your product needs (raw naturalness vs. control and auditability), not by which sounds newer. The realtime architecture and latency budget playbooks walk the decision in detail.
The real-time control layer — the actually-hard part.
Independent of which architecture you pick, a voice agent has to manage the flow of a live conversation. This is where the engineering effort really goes.
- Turn-taking & endpointing. The agent must decide when you have finished speaking (endpointing) and when it should speak. Too eager and it cuts you off mid-sentence; too slow and it feels laggy and dead. Modern systems have moved beyond raw silence detection toward semantic / model-based turn detection that considers whether your sentence actually sounds complete, not just whether you paused.
- VAD (voice activity detection). A lightweight classifier that labels each chunk of audio as speech vs silence. It gates when audio is sent to STT and helps locate turn boundaries — but VAD is not full turn detection. Pause-based VAD alone is a weak turn-taker, because a thoughtful human pause is not the end of a turn.
- Barge-in (interruption). Table stakes. When you start talking over the agent, the system must stop TTS playback and cancel the in-flight LLM/TTS generation within a very short budget, then yield the floor and start listening. An agent you cannot interrupt does not feel like a conversation — it feels like a hold recording.
These three problems are distinct from each other and from the model itself, and getting them right is most of what separates a demo from something people will actually talk to. The turn-taking & barge-in playbook goes deep on tuning them.
The stack in practice.
Voice is a modality that forces a streaming architecture — audio flows continuously in both directions, not in polite request/response turns.
- Transport. Audio streams over WebRTC (for browser and mobile app clients) or WebSocket (for server-to-server links), continuously. Telephony — connecting to actual phone numbers — is typically integrated via SIP.
- Mid-call tools. Function calling happens during the conversation: the agent looks something up or takes an action, then keeps talking — it does not hang up to go do work. This is ordinary tool-calling under a live latency budget.
- Real building blocks (examples, not rankings). For speech-to-speech, the OpenAI Realtime API with the gpt-realtime model (GA August 2025, native audio-in / audio-out). Managed voice platforms include ElevenLabs, Vapi, and Retell AI. Open-source orchestration frameworks include Pipecat and LiveKit Agents. For a cascade you assemble parts: STT via Deepgram or the open-source Whisper; TTS via Cartesia or ElevenLabs.
Once you have the mental model — latency as the master constraint, the cascade-vs-speech-to-speech tradeoff, and the turn-taking / barge-in control layer — the implementation details live in the Voice & Realtime Agents playbooks: realtime architecture, latency budget, turn-taking & barge-in, and the speech stack.