Evaluating Voice Agents

The transcript is the artifact that hides the failure.

Almost every voice team starts by evaluating the text: feed the transcript into the same harness that scores the chat agent, check the answers, ship. That harness measures the one layer least likely to be broken. The model reasoning over clean text is the mature part of the stack; the failures live above and below it.

• Recognition errors are erased by definition. If speech-to-text heard "Preston" as "Preston Road", the transcript contains the wrong string and the agent's answer is a perfectly correct answer to the wrong question. The transcript eval scores it as a pass, because the transcript is the only ground truth it has.
• Timing is not represented at all. A transcript has an order but no clock. The three seconds of silence while a tool ran, the response that started before the caller finished, the barge-in the agent talked over — none of it survives the render, and all of it is what a caller remembers.
• Prosody carries meaning the text does not. "Yeah, sure" is agreement or sarcasm depending on nothing that reaches the transcript. The escalation cue that a human agent catches instantly is a caller's tone changing, and no text pipeline sees it.
• The failure mode is invisible rather than loud. Nothing errors. The agent is fluent, the transcript reads well, the eval is green, and the containment rate quietly sits ten points below what the demo suggested.

The consequence is structural: the unit of evaluation for a voice agent has to be an audio file, and everything downstream of that decision follows from it. See STT, TTS and speech-to-speech for what the recognition layer actually does to your input.

Word error rate is the wrong number; score the entities that decide the outcome.

WER is the metric the speech industry hands you, and it is nearly useless for an agent because it weights every word equally. A transcript can be 96% accurate and still have destroyed the call, if the 4% was the account number. Conversely a transcript full of filler-word errors can be functionally perfect.

• Define an entity set per use case. Names, dates, amounts, order numbers, postcodes, addresses, product SKUs, yes/no answers to consequential questions. These are the tokens on which the task turns. Score entity error rate — the fraction of those that were transcribed wrongly — and treat it as the accuracy metric of record.
• Alphanumerics are their own category and they are the hard one. Reference codes, licence plates, membership IDs: no language model prior helps, telephony bandwidth removes the acoustic distinctions between letters, and a single character is the difference between the right account and someone else's. If your task involves them, they need a dedicated slice of the eval set and probably a dedicated confirmation strategy.
• Score the recovery, not just the error. Recognition will fail; the question is whether the agent noticed. An agent that reads back a captured value and catches a mishearing has turned an error into a two-second delay. One that proceeds confidently has turned it into a wrong outcome. Make "confirmed before use" an assertion in the suite.
• Segment by caller, not just overall. Aggregate entity error rate hides that the agent works for one accent group and fails for another. This is the metric where an average is most likely to be politically comfortable and operationally false.

Build the golden set from real audio, in the conditions the call actually happens in.

The eval set is the whole exercise, and for voice it is materially harder to assemble than for text because the input has physical properties. A clean studio recording of a cooperative speaker reading a script tests almost nothing that production will do to you.

• Start from recorded production calls, with consent and redaction handled first. Real callers interrupt, change their minds mid-sentence, put you on speakerphone, and answer a different question than the one asked. No synthetic script produces that distribution.
• Cover the acoustic axes deliberately. Landline and mobile, speakerphone, car noise, a café, a baby, a bad connection with packet loss. Telephony audio is narrowband — 8 kHz changes every benchmark — so an eval set recorded through a laptop microphone is measuring a channel you do not serve.
• Cover the speaker axes deliberately. Accents, speech rate, age, non-native speakers, and the caller who is angry. Under-representing any of these means shipping an agent that works for the people who happen to sound like your team.
• Keep a hard slice and report it separately. The twenty calls that are genuinely difficult — heavy noise, an unusual name spelled out, a mid-call correction — will be swamped in an average. They are the ones that predict production complaints.
• Use synthetic audio to scale, not to certify. TTS-generated test callers are useful for regression breadth and cost almost nothing, but they are too clean, too well-paced and too cooperative. A suite that is entirely synthetic will report a number your real callers do not recognise. Anchor it: keep a fixed human-audio slice and check that the two move together.

Score the timing, because to the caller the timing is the product.

In text, latency is a comfort issue. In a phone call it is a correctness issue: silence past a certain length is interpreted as the line dropping, and a response that begins too early reads as interruption. These are measurable, they belong in the suite, and they are the scores most teams never take.

• Time to first audio, per turn, at the tail. The p95 is the number callers experience as "the agent is broken", and it hides comfortably behind a healthy median. Break it down against the latency budget so a regression names its own stage.
• Endpointing accuracy in both directions. Cut-off rate — the agent started while the caller was still speaking — and hang time — the caller finished and nothing happened. These trade against each other, so a single "endpointing quality" number is not actionable; you need both, and a stated target for each.
• Barge-in success rate. The caller interrupts: did the agent stop within a couple of hundred milliseconds, and did it correctly incorporate what was said? An agent that talks over an interruption is the single most reliable way to make a caller ask for a human. See turn-taking and barge-in.
• Dead-air incidents, counted, not averaged. Any gap over roughly two seconds with no speech and no filler is an event. Count them per call and alert on the count, because one four-second silence is worse than forty extra milliseconds everywhere.
• Assert on tool-call gaps specifically. The most common source of dead air is a synchronous tool call with no preamble. That is a design bug your suite can catch deterministically: every tool call over a threshold must be preceded by speech.

Judge the outcome, not the conversation.

Once the audio path is instrumented, the remaining question is whether the call did its job — and the temptation is to score how pleasant the exchange sounded, which is the voice equivalent of grading a coding agent on how confident its explanation was.

• Task success is a business-side fact, checked in the system of record. Was the appointment actually booked, at the right time, for the right person? Was the address updated to the string the caller said? Verify against the database, never against the agent's own claim to have done it. This is the same discipline as outcome vs trajectory evaluation, and voice makes the shortcut especially tempting because the summary sounds so plausible.
• Containment is a metric with a dark side. The fraction of calls resolved without a human is the number leadership will ask for, and it improves when the agent refuses to transfer. Always pair it with an escalation-quality score and with what happened after: a contained call followed by a callback within 24 hours is a failure the containment number reports as a success.
• Score the handoff explicitly. When the agent escalates, did the human receive the context, or does the caller repeat everything? A transfer that loses state is worse than not having the agent, and it is invisible in every metric above.
• Use an LLM judge on the transcript for what a judge is good at — tone, policy adherence, whether a required disclosure was made — while keeping the audio-derived metrics separate. A judge scoring a transcript is scoring the layer step 1 told you was already fine; useful, but never the headline. See LLM-as-judge for agents.
• Compliance items are assertions, not scores. Recording disclosure, identity disclosure on outbound calls, consent capture: these are boolean and they are legal. They belong in the suite as invariants that block a release, alongside the rest of what outbound calling obliges you to say.

Make it a harness, not an exercise.

A voice eval that runs when someone remembers is a voice eval that runs before the first launch and never again. The distinctive operational problem here is that your stack has more moving vendors than a text agent — recogniser, model, synthesiser, telephony carrier — and any of them can change without telling you.

• Replay audio through the real pipeline, not through a mock. The point is to exercise the recogniser and the endpointing logic. A harness that starts from text has already skipped the two components most likely to have regressed.
• Run the fixed suite on a schedule, not only on deploy. An STT vendor's model update, a carrier changing its codec, a TTS voice being revised: none of those appear in your change log, and the scheduled run is what catches them the same day.
• Keep a live-traffic sample scored continuously. A small percentage of production calls, human-reviewed on the entity fields and the timing events, indefinitely. This is where you find the caller population your golden set does not represent.
• Treat every escalation as a labelled example. "The caller asked for a human" is free supervision arriving daily. Cluster the reasons and the top three will name your next three fixes more accurately than any offline analysis.
• Version everything acoustic. Recogniser model and version, voice ID, VAD settings, endpointing thresholds, codec. When a metric moves you need to attribute it, and in voice the change is at least as likely to be in the audio configuration as in the prompt.

Related: voice agent failure modes for the catalogue this suite is trying to detect, realtime agent architecture for why the cascade and speech-to-speech designs need different instrumentation, and evaluating agents for the general case.