Word error rate is the number every speech-to-text page leads with and the number that will change your voice agent the least. The slice of a turn that decides whether the conversation feels human is end-of-turn detection — the moment the system decides you stopped talking — and it is several times larger than the transcription latency underneath it. You are choosing a turn detector that happens to transcribe.
At a glance
Four providers a voice-agent team will actually shortlist in 2026, and the axis each one optimised.
| Provider | Streaming model | Turn detection | Deployment |
|---|---|---|---|
| Deepgram | Flux (conversational ASR); Nova-3 for general streaming | Native to the recogniser — emits turn events | Cloud, plus self-hosted for enterprise contracts |
| AssemblyAI | Universal Streaming (universal-3-5-pro on the v3 socket) |
Semantic plus acoustic endpointing, silence-based fallback | Cloud |
| ElevenLabs | Scribe v2 Realtime | Not provided — you bring voice-activity detection | Cloud |
| Speechmatics | Ursa 2, with the Flow voice-agent API on top | A silence threshold you configure | Cloud and on-premise |
The latency that decides the conversation is not the one on the pricing page
Vendors publish first-partial latency and time-to-final because those are properties of the recogniser. Neither is what a caller experiences. A caller experiences the gap between the last syllable they spoke and the first syllable they hear back, and that gap has four parts in series.
The first slice dominates, and it dominates by construction. Silence-based endpointing has to wait long enough that a mid-sentence pause is not mistaken for a finished thought — Speechmatics' own guidance suggests 1.5 seconds as a reasonable starting point for voice-agent use, with sub-second finals possible only under deliberate tuning. Meanwhile AssemblyAI's Universal-3 Pro Streaming posted a 307 ms P50 latency in a Hamming.ai benchmark across more than four million production calls, against 516 ms for Deepgram's Nova-3. That 209 ms difference is real, and it is a quarter of the endpointing wait sitting on top of it.
Which means a team that switches provider to save 200 ms of transcription, while leaving a 1.5-second silence timer in place, has optimised roughly a seventh of the problem. The latency budget for a voice agent is won or lost at the turn boundary.
Word error rate is close to settled, and the remaining errors are not where you think
Published English WER for the streaming tiers of all four sits in a band of a few points — the Hamming.ai run put AssemblyAI at 8.14% and Deepgram Nova-3 at 9.87% on production call audio, and independent English benchmarks put ElevenLabs Scribe near the top of the accuracy tables. A point or two of WER is roughly one word in a hundred, and in a voice agent most of those words are function words the intent classifier ignores.
The errors that break an agent are concentrated in a narrow class: proper nouns, account numbers, drug names, SKUs, addresses, and anything spelled letter by letter. Those are the tokens that become tool-call arguments, and a single wrong digit is not a 1% quality regression — it is a wrong transfer. Every one of these providers supports biasing toward a custom vocabulary, and configuring that list well moves entity accuracy far more than switching vendor moves aggregate WER.
Build your evaluation set out of the utterances that produce tool calls, not out of general conversation. A provider that is 1.5 points worse overall and 8 points better on your product's SKU list is the better provider, and no public leaderboard will tell you that.
Multilingual is the one accuracy axis where the spread is still wide. ElevenLabs reports 93.5% accuracy on FLEURS across 30 languages with coverage of 90-plus, and Speechmatics has built its reputation on breadth across 55 languages and on code-switching — a caller moving between two languages inside one sentence. If your callers do that, the accuracy gap between providers stops being a rounding error.
Turn detection is the actual product
Where each one puts the decision
The four differ structurally, not by a tuning constant. Deepgram's Flux folds turn detection into the recogniser itself, using acoustic and semantic signal together to emit StartOfTurn, EagerEndOfTurn, TurnResumed and EndOfTurn events; the vendor reports end-of-turn within 1.5 s at p95 and no separate voice-activity-detection layer. AssemblyAI keeps the two components separate but ships intelligent endpointing that combines acoustic and semantic features with a silence-based fallback, which is the pragmatic middle. ElevenLabs optimised for something else entirely — roughly 150 ms first-partial latency and predictive streaming across 90-plus languages — and leaves the turn decision to your own VAD. Speechmatics gives you a threshold and expects you to tune it.
The consequence is a different debugging surface in each case. With Flux, a mistimed interruption is a model behaviour you influence through eagerness settings and cannot inspect directly. With AssemblyAI or a bring-your-own-VAD stack, it is your code, your threshold, and your logs — slower to get right and much easier to reason about at three in the morning. Neither is strictly better; they fail in different places, and you should pick the failure you can staff.
The failure mode nobody benchmarks
Cutting in too early and waiting too long are not symmetric errors. A half-second of extra silence reads as a slightly slow agent. Interrupting a caller mid-thought reads as rude, and it costs a full recovery turn — the caller repeats themselves, the agent apologises, and you have spent four seconds to save eight hundred milliseconds. Deepgram reports Flux cutting interruptions by around 30% relative to a conventional ASR-plus-VAD pipeline, which is the right thing to measure and the thing almost no comparison table reports.
Barge-in is the mirror image and it is separate work regardless of provider: detecting that the caller has started speaking over the agent, stopping synthesis immediately, and deciding what the agent had already said out loud. See turn-taking & barge-in for the state machine that sits behind all of this.
Price is the wrong axis, and here is the arithmetic
List rates for streaming English land roughly between $0.15 and $0.50 per audio hour across these providers — AssemblyAI has priced aggressively at the low end, Deepgram's Nova-3 English streaming sits near the top of that band, and ElevenLabs Scribe falls in between depending on plan. Speechmatics prices its Flow voice-agent API per minute, which bundles more than transcription and is not comparable line-for-line.
Now put that next to the rest of a voice agent. An hour of conversation is perhaps 8,000 spoken words in each direction plus the system prompt, the tool definitions and the transcript re-sent on every turn — the quadratic that agent cost control describes. Text-to-speech is usually billed at a comparable or higher rate than speech-to-text. In most deployments the recogniser ends up the smallest line on the bill.
Spend the negotiation on the contract terms that are hard to change — data residency, retention, whether audio is used for training, and whether an on-premise deployment exists at all — rather than on the per-hour rate. Those are the terms that will decide whether you can keep the provider when compliance reviews the design. See data residency & sovereignty.
When to pick which
| Use case | Pick | Because |
|---|---|---|
| Real-time phone agent, English-dominant, latency is the product | Deepgram Flux | Turn detection is inside the model, so the largest slice of the turn budget is a vendor problem rather than yours. |
| Voice agent plus post-call analytics on the same audio | AssemblyAI | Competitive streaming latency and WER, with the richest audio-intelligence suite behind the same account. |
| Consumer product across many languages, agent turn logic already built | ElevenLabs Scribe v2 Realtime | Broadest language coverage and the fastest partials; you already own the VAD it does not ship. |
| Regulated deployment, on-premise, or heavy code-switching | Speechmatics | On-premise is a hard requirement or it is not, and it eliminates the other three. Code-switching is its long-standing strength. |
Whichever you pick, run the shortlist against your own recorded calls before signing. The evaluating voice agents playbook covers the harness; the short version is that a hundred of your own calls will rank these four differently from every public benchmark, and your hundred is the ranking that pays.
FAQ
Is word error rate useless for choosing a speech-to-text provider?
Not useless, but close to settled between the leading streaming models — a point or two of aggregate WER is around one word in a hundred, and in a voice agent most of those are function words. Entity accuracy on names, numbers and SKUs is the measurement that predicts whether tool calls come out right, and it varies far more between providers than the headline figure does.
What is end-of-turn detection and why does it dominate latency?
It is the decision that the speaker has finished, and it sits in series before transcription, model inference and synthesis. A silence-based detector must wait long enough not to mistake a mid-sentence pause for a finished thought, which puts it in the several-hundred-milliseconds to 1.5-second range — typically the largest single slice of the gap a caller perceives.
Can I use Whisper or an open-weights model instead?
For batch transcription, often yes. For a live agent the gap is not accuracy but turn-taking: an open-weights recogniser gives you a transcript and leaves endpointing, barge-in and partial-result handling entirely to you, which is the expensive part of the build rather than the cheap part.
Does the speech-to-text choice affect my agent's cost materially?
Rarely. Streaming English list rates cluster between roughly $0.15 and $0.50 per audio hour, which is usually the smallest line on a voice agent's bill next to language-model tokens and speech synthesis. Optimise the contract terms and the turn latency instead.
Should turn detection live in the recogniser or in my application?
In the recogniser if you want the vendor to own the hardest latency problem and you can live with tuning it through settings rather than code. In your application if you need to inspect and reproduce every turn decision — for example in a regulated call flow where you must explain why the agent interrupted.
Further reading
On this wiki:
- The speech stack — how ASR, the model and TTS fit together, and where each one can be swapped.
- Latency budget — the full accounting for a voice turn, target by target.
- Turn-taking & barge-in — the state machine behind interruption handling.
- Multilingual voice agents — code-switching, language detection, and what breaks per language.
- Voice & realtime agents — the concept-level introduction.
- ElevenLabs vs Vapi vs Retell vs OpenAI GPT Realtime — the platform layer above this one.