Playbooks / Voice & Realtime Agents

Voice & Realtime Agents

Realtime voice agents — speech stack, turn-taking, barge-in, latency budgets, voice-specific tooling and state.

  1. Realtime Agent Architecture
    The session’s context window is a clock: streamed audio fills it at a documented ~25 tokens per second against 32k on most Live models and 128k on native-audio ones, which is where the published 2-minute and 15-minute caps come from. Three places to put the speech boundary — full cascade, half-cascade, native audio — plus a transport that expires before the session does, and why a heavily-tooled voice agent still ends up a cascade.
  2. The Latency Budget
    Sum the latency figures the voice vendors published through 2026 and one turn reaches about 1,100 ms before a packet crosses the internet — which is how a stack of uniformly fast components ships at twice the 200 ms gap humans expect. Two of the seven lines are the ones nobody publishes: endpointing, which a fixed timer charges to every turn, and the transport plus playout path. Budget roughly twice any vendor component figure, cut hops rather than shop for speed, and keep the two clocks apart — response latency is a perception cost, call duration is a cash cost, and the τ-Entity confirmation scaffold's 21–28 seconds is a disaster on one and a rounding error on the other.
  3. Turn-Taking & Barge-In
    The turn manager is the only component in a voice stack holding mutable state, and a barge-in invalidates it mid-write: generated, synthesised and actually-heard diverge by seconds of speech, so a history that stores what the model produced leaves the agent referencing sentences nobody heard. Replace the silence threshold with a completion model and expose the operating point, then build the reconciliation path — truncate to confirmed playback, bias toward ‘not heard’ because the two errors cost differently, set an explicit truncated flag, and cancel the stream before a discarded tool call runs. Full duplex deletes the boundary and makes the bookkeeping harder.
  4. STT, TTS & Speech-to-Speech
    The cascade-or-native choice is not about voice quality: in a cascade every control you own — redaction, guardrails, read-back, the transcript an auditor reads — runs inline on text, and that text is the model's own input. Native removes the transcription-error tax and with it the layer those controls ran on, so each becomes a lagging observer of audio and your record becomes a reconstruction by a second ASR pass with its own errors — the agent can act correctly while the evidence says otherwise, and nothing compares them. Score entities not words, normalise text before synthesis, and reconcile the agent's structured values against the side transcript.
  5. Tool Use & State in Voice
    Calling tools without dead air: preambles, async/parallel tool runs, confirm-by-ear before mutating, and slot state across an interruptible call.
  6. Voice Agent Failure Modes
    Hallucinated hearing, dead air, the infinite apology loop, the latency death spiral, and the escalation/handoff you must design for.
  7. Outbound voice agents
    Agents that **make** the call instead of answering it — pacing, abandonment, identity disclosure, and the regulatory landmines that turn a clever demo into a fine.
  8. Evaluating Voice Agents
    Transcript evals score the one layer that was not broken: build the golden set from recorded audio, measure entity error rate rather than WER, and treat timing as a first-class score.
  9. Telephony & PSTN Integration
    Half the turn latency, all the audio quality and whether the call connects at all live in a carrier path you cannot profile: the fixed transport tax, your number as a reputation asset, the missing metadata channel, and consent as a code path rather than a prompt.
  10. Multilingual & Code-Switching Voice Agents
    Adding a language breaks recognition, not generation — and per-turn language lock, the standard fix, is exactly what a bilingual caller violates in their first sentence, with the damage landing on the names and numbers you were about to pass to a tool.
  11. Caller Authentication for Voice Agents
    Three seconds of audio clones a customer and roughly one in five biometric fraud attempts is now a deepfake, so a voiceprint identifies but no longer authenticates — move the proof out of the audio channel, bind it to the action rather than the call, and notice that your own outbound agent is normalising the attack.
  12. Retrieval Inside the Voice Turn
    A grounded answer has to start leaving the speaker about 800ms after the caller stops, and a retrieve-rewrite-rerank chain spends most of that before the model sees a document — so retrieval latency is an accuracy metric: start querying on the partial transcript, precompute the head of the question distribution, and delete the stages you were told were mandatory.
  13. Escalation & Warm Transfer
    A voice agent can nail every turn and still lose the customer in the handoff, because the human answers as if the call never happened — so the handoff is judged by how much the caller repeats, not by whether it connected: escalate before the caller asks, carry a context packet (verified identity, the caller’s own words, actions already taken) that lands before the human’s first word, always design the no-human branch, and page yourself on repeat rate rather than transfer rate.
  14. After-Call Work & CRM Writeback
    The caller hears the conversation once; the disposition code and summary are read for years by routers, analysts and auditors — and your voice eval stops when the caller hangs up. The disposition is a classification problem wearing a generation problem’s clothes, your wrap-up taxonomy is probably already broken, and the summary must quote rather than characterise, because ASR errors concentrate exactly on the names and amounts a durable record cannot get wrong.
  15. Recording, Consent & Redaction
    Your retention rule points at the call recording, and the recording is now the least interesting copy: the same minute also lives as a transcript, a model context, tool arguments, a trace span and a CRM summary — five artefacts the agent created that inherited no policy. Gate the buffer on consent rather than call setup, keep card data off the agent leg entirely because a model cannot look away, and run one versioned redactor in front of every sink at write time.
  16. Replacing an IVR
    The two artefacts you start from are both traps: the menu tree records what touch-tone could express rather than what callers want, and containment — quoted at 5–10% for legacy IVRs against 60–90% for voice agents — scores the caller who gave up as a success. Build the intent inventory from the zero-out transcripts, migrate one intent at a time in front of the IVR you already trust so rollback is a config flip, measure resolution without a callback in 72 hours with in-agent abandonment as the veto, and keep DTMF for digits.
  17. Accessible Voice Agents
    Your agent does not have a failure rate, it has one per kind of voice, and the cohorts that fail hardest — disordered or slow speech, strong accents, older callers, anyone on a relay service — have the fewest alternatives, so their failures arrive as hang-ups and never reach your metrics. Silence-based endpointing is not merely blind to them, it is the mechanism: stratify by speech rate rather than by any label, raise the threshold permanently after the first re-prompt, keep DTMF and a human path live at every turn, and gate releases on worst-cohort over median-cohort success.
  18. Alphanumerics Over Voice
    Your transcription is excellent and your order lookups fail, because a booking reference has no language model behind it and exact match is per-character accuracy raised to the length — 97% per character is 73.7% on ten characters. Provider spelling modes and keyword biasing buy percentage points; what changes the shape is resolving against the small candidate set the caller's phone number already gives you, chunked capture with per-chunk readback when you truly must, confirmation set by consequence, and taking the finding upstream to the identifier format.
  19. Disclosing the Agent on a Call
    The one sentence you are legally required to say is the one most likely to be cut by your own barge-in, and it logs as played. Three obligations with different shapes — EU AI Act Art. 50 up front since 2 August 2026, Utah on request, Utah again prominently for high-risk work — need three code paths: a short non-interruptible opener logged on final frame, a must_disclose predicate re-evaluated at transfer and party-join and resume, and a deterministic "are you a bot" intent with a constant-string answer, because a system-prompt line makes statutory compliance a model behaviour.
  20. Card Payments Over Voice
    Every contact-centre descoping technique works by removing a listener from the audio path, and a voice agent is not a listener — it is the call, so one spoken card number lands in six artefacts at once, two of which PCI DSS says may never be stored post-authorisation. Three architectures keep the data out of the model and they differ only in what they cost the caller. The change to make today is the tool signature: if a PAN can be an argument, your trace store is in scope.
  21. Emergency & Urgent Calls
    No standard tells a voice agent where to route a crisis call, but Kari's Law and dispatchable location already attach to your phone system — so put a deterministic matcher on partial ASR below the model, blind-transfer with a recorded fallback, fix the miss rate before precision, and refuse triage, first-aid instructions and being the crisis line.
  22. Voicemail & Answering-Machine Detection
    Twilio's own FAQ puts answering-machine detection at about four seconds after answer on default settings, and the US telemarketing rule calls a call abandoned if no live representative is connected within two seconds of the called party's completed greeting, capped at 3% of live-answered calls per 30-day campaign — so gating the first word on a verdict is non-compliant by construction, and it spends that silence on the one dial in ten a human answered. Speak on answer with detection as a concurrent event stream, write a four-second opener that works as a truncated voicemail, and treat a voicemail drop as a second permission: the FCC ruled in November 2022 that ringless voicemail to a wireless number is a call needing consent.
  23. Multi-Party Calls
    “Let me put you on with my wife” is a complete bypass of your identity controls using no technique at all, because you verified a call and a call is not a person. Three 2026 benchmarks agree the models cannot help: MP-Bench puts realtime voice agents at or below 22% on multiparty comprehension and near chance on multiparty turn-taking. Streaming diarization revises speaker labels after you have acted on them, so bind every consequential action to a party re-verified on this utterance, raise party-join as an interrupt, and make UNKNOWN fail closed.
  24. Audio Quality & Codecs
    An 8 kHz call cannot carry sound above 4 kHz, and the fricatives that separate S from F live above it — so every recognizer has the same blind spot on a PSTN leg and no model closes it. Twilio Media Streams hands you G.711 u-law at 8 kHz and OpenAI's Realtime telephony format is fixed there too, so the usual stack is narrowband end to end; upsampling to 24 kHz satisfies an input contract and adds nothing. The path runs at its narrowest hop, the one cited cross-band WER figure is from 2014, and the fix is to put a codec column in your eval set.
  25. Long Lists Over Voice
    Speech has no scrollback and Cowan's four-chunk estimate leaves a caller holding the option they are hearing and maybe the one before it, so six options is twenty-five seconds of monologue whose only reliable output is a recency bias your conversion data will read as a preference. The defect is the tool signature, not the prompt: a tool that can return a result set forces the model to truncate it by position, uninstrumented. Return an answer, a discriminating question with counts, or a too-many outcome; lead with the discriminator; never speak an index; and alert on the longest uninterrupted agent utterance.
  26. Appointment Scheduling Over Voice
    A voice agent double-books before it mishears anything: speaking three options is 9.6 seconds at 150 wpm, and the window from reading availability to an accepted write is 12 to 18 seconds, during which 40 offers an hour put one in six on the same slot — because every agent offers the earliest first. Hold one slot before you speak it (three held per call takes 6.7 slots out of circulation at 400 offers an hour; one takes 2.2), confirm with If-Match so a moved calendar returns 412 instead of a second booking, and write the recovery sentence before you need it. Instrument hold-expiry, precondition-failure and spoken-versus-booked mismatch separately, because only the third catches silent substitution.
  27. Calling Another Company's IVR
    Dialling out into someone else's phone tree makes speech the smallest part of the job: the top-level loop is a state classifier — menu, hold, human, voicemail, dead air — because hold music and a half-heard menu transcribe to nothing you can act on. Send digits as RFC 4733 named telephone-events rather than synthesising tones into the audio path, where a lossy codec smears them and packet loss turns one press into zero or two. Hold time is unbounded, not yours to control, and billed per second of wall-clock on a native audio stack, so silence costs what speech costs: demote the model while waiting, cap the wait, and schedule against observed queue length.