Voice & Realtime Agents

Realtime Agent Architecture

Cascade (STT→LLM→TTS) vs native speech-to-speech, the stateful audio transport, and the one decision everything else hangs on: where the agent loop lives.

The Latency Budget

The sub-second turn accounted for line by line: where the milliseconds go, why endpointing is the biggest slice, and perceived vs actual latency.

Turn-Taking & Barge-In

VAD vs endpointing, semantic end-of-turn detection, mandatory barge-in, echo cancellation as a prerequisite, and backchannels vs real interruptions.

STT, TTS & Speech-to-Speech

Streaming STT, the transcription-error tax, TTS time-to-first-audio, native audio models, and why 8 kHz telephony changes every benchmark.

Tool Use & State in Voice

Calling tools without dead air: preambles, async/parallel tool runs, confirm-by-ear before mutating, and slot state across an interruptible call.

Voice Agent Failure Modes

Hallucinated hearing, dead air, the infinite apology loop, the latency death spiral, and the escalation/handoff you must design for.

Outbound voice agents

Agents that **make** the call instead of answering it — pacing, abandonment, identity disclosure, and the regulatory landmines that turn a clever demo into a fine.

Evaluating Voice Agents

Transcript evals score the one layer that was not broken: build the golden set from recorded audio, measure entity error rate rather than WER, and treat timing as a first-class score.

Telephony & PSTN Integration

Half the turn latency, all the audio quality and whether the call connects at all live in a carrier path you cannot profile: the fixed transport tax, your number as a reputation asset, the missing metadata channel, and consent as a code path rather than a prompt.

Multilingual & Code-Switching Voice Agents

Adding a language breaks recognition, not generation — and per-turn language lock, the standard fix, is exactly what a bilingual caller violates in their first sentence, with the damage landing on the names and numbers you were about to pass to a tool.

Caller Authentication for Voice Agents

Three seconds of audio clones a customer and roughly one in five biometric fraud attempts is now a deepfake, so a voiceprint identifies but no longer authenticates — move the proof out of the audio channel, bind it to the action rather than the call, and notice that your own outbound agent is normalising the attack.

Retrieval Inside the Voice Turn

A grounded answer has to start leaving the speaker about 800ms after the caller stops, and a retrieve-rewrite-rerank chain spends most of that before the model sees a document — so retrieval latency is an accuracy metric: start querying on the partial transcript, precompute the head of the question distribution, and delete the stages you were told were mandatory.