STT, TTS and speech-to-speech: the choice is where your controls live.
Pick a cascade or a native audio model on voice quality and you will discover, during your first compliance review, that you chose something else entirely: in a cascade every control you own — redaction, policy checks, read-back confirmation, the transcript an auditor will read — operates on text, and the text the model acted on is the same object you kept. Go native and the text layer leaves the critical path, so each of those controls becomes an asynchronous observer of audio and your record becomes a reconstruction. The transcription-error tax is real and a native stack genuinely avoids it; what it buys with that saving is a system where your evidence and your model's input are two different artefacts and nothing tells you when they disagree.
Step 1Streaming STT is not an optimisation. It is the only mode an agent can consume.
Batch STT transcribes a finished clip with maximum accuracy. Streaming STT emits partial hypotheses while the caller is still talking and revises them as more audio arrives. For an agent, the second one is not a trade-off you evaluate; it is what makes endpointing, speculative tool calls and retrieval inside the turn possible at all. You accept a slightly worse final transcript to buy a latency budget you otherwise cannot meet.
- Partials are unstable by design. "I want to" becomes "I want to cancel" becomes "I want to cancel my… upgrade". Never let a side effect fire on a partial. Gate every write, charge and transfer on the finalised transcript — the same rule, for the same reason, as never acting on a streamed tool argument.
- A published word error rate transfers only if the corpus does. Every WER figure in this market is measured on a corpus, and yours is a narrowband phone line carrying your customers' surnames, SKUs and accents. Vendors publish batch numbers on clean read speech; your agent consumes streaming partials on 8 kHz audio. Those are different quantities and the ranking between two vendors can invert between them, which is why the only number worth acting on is one you measured — see evaluating voice agents for how to produce it, and audio quality and codecs for why the channel caps everybody equally.
- Measure entities, not words. A transcript that is 97% accurate per word and wrong on the account number is a failed call with a good score. Score the fields you are going to put into a tool call.
Step 2The transcription-error tax, priced properly.
This is the defining liability of any cascade, and it is not an accuracy problem — it is a type problem. STT does not return "I'm not sure". It returns its best guess as confident text, and the model then reasons over that text as ground truth. Uncertainty that existed in the audio is destroyed at the boundary, and nothing downstream can recover it.
# the tax: confident text from uncertain audio caller said: "don't ship it to the old address" STT produced: "do ship it to the old address" LLM reasons: confidently. wrong order. no flag raised. # one phoneme, one negation, one shipment. the model was # not wrong about anything it was given.
Notice which words are expensive. The tax does not fall evenly across a sentence; it concentrates on exactly the tokens that have no language model behind them and the ones that invert meaning — digits, proper nouns, booking references, and negations. A model can repair "I'd like to to cancel" without being told. It cannot repair a dropped "don't", because the repaired sentence is also fluent.
Do not let the model sort it out. On names and numbers, "sort it out" means guessing plausibly and confidently wrong. For high-stakes slots — amounts, identifiers, yes/no, addresses — read the value back and get explicit confirmation. The read-back is not politeness; it is the error-correcting code for a lossy channel, and alphanumerics over voice is the arithmetic for why per-character accuracy raised to the length of an identifier makes it mandatory.
Step 3TTS: three numbers, and only one of them is quality.
Naturalness stopped being the bottleneck somewhere around 2025, and teams keep shopping on it. The agent-relevant axes are elsewhere:
- Time to first audio. The only TTS latency figure that touches perceived responsiveness. A beautiful voice at 800 ms to first sample is unusable in conversation; a plainer voice at 150 ms is not. Budget it as a line in the latency budget and measure it on your own transport, not from a datasheet.
- Incremental synthesis. The engine must accept text as it arrives and emit audio as it goes, so you start speaking on the model's first clause instead of its last token. Without it, your TTS latency is not time-to-first-audio — it is the model's full generation time plus time-to-first-audio, and no amount of synthesis speed fixes that.
- Clean interruption. Barge-in has to cut mid-utterance, and your turn manager has to know how much audio was actually delivered. An engine that buffers whole sentences is an engine whose "played" and "heard" diverge by seconds.
And one thing that is yours rather than the vendor's: text normalisation. "$1,204.50" and ACC-0042 and "2026-11-02" must be spoken correctly every single time, which means you normalise before synthesis rather than hoping the model's prosody guesses right. Write it as a tested function, not a prompt instruction. It is the cheapest reliability win in the stack and the one most often left to chance.
Step 4Native speech-to-speech removes the tax by removing the text layer your controls run on.
A native audio model — OpenAI's gpt-realtime, Gemini Live — hears prosody and emotion directly and speaks expressively, with no transcription step on the critical path and two fewer serial hops of latency. Everything in Step 2 goes away. The caller's negation reaches the model as sound, and a dropped "don't" is no longer a failure mode you can have.
On instruction-following the gap is closing but real: OpenAI reports gpt-realtime at 30.5% on the MultiChallenge audio instruction-following benchmark, up from 20.6% for its December-2024 predecessor. Read that as what it is — rapid progress, and audio-native instruction following still trailing text. Cost-sensitive paths can drop to a mini tier and trade headroom for price.
Here is what is actually being traded, and it is not voice quality. Every control you built in the cascade was a text operation:
control cascade native
--------------------- ---------------------- ---------------------
PII redaction regex/NER on transcript audio, async, post hoc
policy / guardrail text classifier inline audio classifier, lagging
read-back confirm you compose the string model improvises it
retrieval grounding transcript is the query need a parallel ASR pass
the audit record the model's own input a separate transcription
Each row is work you now own. None of it is impossible — audio classifiers exist, parallel ASR is cheap, and a native model can be instructed to confirm values. But every one of them moves from "inline and blocking" to "alongside and lagging", which changes what a guardrail can promise. A text classifier can refuse before the words leave the speaker. An audio classifier watching the output stream can only tell you what was already said. If your deployment is one where card data or consent and redaction decide the architecture, that difference is the architecture.
Step 5In a native stack, your evidence and the model's input become two different artefacts.
This is the consequence nobody plans for, and it is the one that costs the most to retrofit. You will still produce a transcript, because your CRM, your quality team, your regulator and your after-call writeback all require one. In a cascade that transcript is the model's own input: whatever the agent reasoned over is what you kept, errors included, which is unflattering and extremely useful. In a native stack the transcript is produced by a separate recognition pass over the same audio — and that pass has its own error profile.
# cascade: one artefact, so a disagreement is impossible audio -> STT -> "do ship it to the old address" -> model -> action | +--> record # same string the model used # native: two artefacts, and no one compares them audio -> model -> action # heard "don't" \ +-> side ASR -> record # wrote "do" # the record now contradicts the action, # and the record is what gets read in six months
The worked case is worth sitting with. The native model hears the negation correctly and does the right thing: it does not ship to the old address. The side ASR drops the "don't" and your record says the caller asked for the old address. Six months later a dispute is resolved from the record. The agent behaved correctly and the evidence says otherwise, and no monitor in your stack is positioned to notice, because each component did its job.
- Mark the record as a reconstruction. Store the recognition model and version alongside it, and never let a downstream system treat it as a statement of what the agent processed.
- Reconcile where it is cheap. You do not need full agreement — you need agreement on the fields that matter. Have the agent emit its understood values as structured tool arguments, and compare those against the side transcript on identifiers, amounts and polarity. Disagreement is a reviewable event, and it is a far better signal than any transcript-level score.
- Keep the audio longer than you keep the transcript, subject to consent and retention. In a native stack the audio is the only artefact the model actually saw.
- Score against audio, not text. A transcript eval of a native agent grades a component that was not in the loop. Build the golden set from recordings.
Step 6The honest tradeoff, and how to decide in an afternoon.
A cascade buys inspectability, per-stage substitution and controls that run inline on the same object you keep as evidence, at the price of a permanent transcription-error tax. Native buys expressiveness, the lowest achievable latency and no tax, at the price of opaque audio reasoning, lagging guardrails, and an audit record that is a reconstruction rather than the model's input. There is no stack without a tax — only a tax you chose.
Three questions settle it faster than a bake-off:
- Does any control have to block? If a classifier must prevent something being said rather than report it afterwards, you need text inline, and that means a cascade or a hybrid with a text path for the regulated turns.
- Will the transcript be read by someone who can impose a consequence? If yes, decide now whether it is the model's input or a reconstruction, and write that distinction into the record format rather than into a wiki page.
- Is the task's failure concentrated on entities or on rapport? Identifiers, amounts and addresses push you to a cascade with aggressive read-back. Open-ended conversation where tone carries the task pushes you native.
Do this first, whichever you pick: take twenty real recorded calls from the channel you actually ship on, run them through both configurations, and score only the entities and the polarity — identifiers, amounts, yes/no, negations. You will learn more in an afternoon than from every published benchmark in this market, because you will have measured the one corpus that transfers. Then add the reconciliation check from Step 5 before you have any history to go back and fix.