Multilingual & Code-Switching Voice Agents

The failure is upstream of the model, in the first few hundred milliseconds.

Teams plan multilingual support as a generation problem: can the model answer in Spanish? It can, and that is not the hard part. Every real difficulty sits in the layer before the model, where audio has to be committed to a language before anyone knows what was said.

• Something has to decide which language this is, and it decides early. A cascade stack must pick a recognition target from the opening audio — before enough evidence exists — because streaming transcription cannot wait for the sentence to end without destroying the latency budget.
• A wrong guess is not a slightly worse transcript, it is a different one. Recognition constrained to the wrong language does not return uncertainty; it returns confident, well-formed words from the wrong vocabulary. The downstream model then reasons about a plausible sentence the caller never said, which is the failure mode hallucinated hearing describes — with the language layer supplying the hallucination.
• Accent and bilingualism attack the detector directly. A habitual bilingual speaker's phonetics blend both languages, which is exactly the signal a language identifier keys on. The speakers most likely to code-switch are the speakers the detector is worst at, and that correlation is not a coincidence you can engineer around with a threshold.
• Detect-then-route costs a round trip you do not have. Running identification as a separate pass before recognition adds latency at the front of every turn, where the caller's tolerance is lowest and where the turn-taking machinery is already spending its budget.

Per-turn language lock is the standard fix, and code-switching is what breaks it.

The usual architecture detects a language, locks it for the session or the turn, and gets a large accuracy win for it. It works well for a caller who speaks one language. It fails on the population it was added for.

• Between turns is the easy case. A caller who answers in Spanish and then asks a follow-up in English is handled by re-detecting each turn. Slightly more expensive, structurally fine.
• Within a sentence is the hard case, and it is the common one. Product names, place names, technical vocabulary and numbers routinely stay in a second language inside an otherwise monolingual sentence. A single locked recognition target has no representation for the embedded span and will substitute the nearest thing in its own vocabulary.
• The damage concentrates at the boundary. Errors cluster around the switch point rather than spreading evenly, and streaming decoders can stall or truncate there — which is worse than a plain substitution, because a dropped word leaves a grammatical sentence with a missing constraint.
• An end-to-end multilingual recogniser is the structural answer where you can afford it. Handling language fluidity natively, without a pre-committed routing decision, removes the whole failure class rather than tuning it. The costs are real — usually a larger model, sometimes higher time-to-first-token, and a smaller set of vendors — and that trade is the one to evaluate deliberately rather than defaulting past.
• Where you must route, route on a window and allow revision. Detect over a longer span than the first phoneme, and allow the transcript to be corrected once the evidence arrives. Then make sure the agent does not act on the pre-revision text, which is a real bug in stacks that fire tool calls on partial transcripts.

The words that get lost are the ones you were going to pass to a tool.

This is the argument that should change your evaluation strategy. Aggregate word error rate is a poor guide here because it treats every word alike, and the errors are not distributed alike.

• Entities cluster in the switched span. A caller switches language for exactly the things that do not translate — a person's name, a street, a brand, a model number. Those are the tokens with the highest information content in the utterance and the lowest redundancy, so the recogniser's language model cannot repair them from context.
• A 5% word error rate can coexist with a 30% entity error rate, and the second number is the one that determines whether the tool call succeeds. Measure entity error rate directly, as evaluating voice agents argues, and break it out by whether the entity fell inside a switched span.
• Downstream fluency hides all of it. The model receives a clean-looking sentence, reasons correctly about it, and produces a confident answer about the wrong account. Nothing in the trace looks broken. This is the specific reason multilingual regressions get discovered by customers rather than dashboards.
• Confirm entities by ear before you mutate anything. Reading back the name or the number and getting an acknowledgement is the only cheap control that survives a bad transcript, and it costs one turn. Apply it to the arguments, not to the intent — the pattern is in tool use and state in voice.
• Constrain recognition where the value space is known. If the order ID has a fixed shape, or the city is one of two hundred you serve, biasing or validating against that set recovers most of the loss without touching the recogniser. Free-form recognition of a closed set is a solved problem you are choosing not to solve.

The output side: one voice, one language, and a policy you have to write down.

Synthesis has its own constraints, and they are less discussed because they are less catastrophic — but they determine whether the agent sounds competent to a bilingual listener.

• Most synthesis engines cannot switch language mid-utterance convincingly. A voice built for one language pronounces the other's words with its own phoneme inventory, which is the audible difference between "an agent that speaks my language" and "an agent reading my language badly".
• So you must choose an answer language per turn, explicitly. The default of mirroring whatever the recogniser last decided means a detection error becomes an audible language switch, which is far more jarring to the caller than a slightly wrong word. Prefer stability: mirror the caller's dominant language, and change only on a clear, sustained signal.
• Keep proper nouns in their own language, and accept the accent. A street name or brand rendered in the caller's language is often unrecognisable to them. Mixed pronunciation of a name inside an otherwise monolingual sentence is what human bilinguals do, and it is the right target even when the engine does it imperfectly.
• Use one voice per language and keep it stable across the call. Switching timbre mid-conversation reads as a transfer to a different person and resets the caller's model of who they are talking to.
• Localise the disclosure, not just the content. Whatever your jurisdiction requires the agent to say about being automated has to be said in the language the caller actually speaks, at the start, and it has to be in your test suite for every supported language — see outbound voice agents for how much weight that sentence carries.

Numbers, dates and addresses are a separate normalisation problem per language.

Even with a perfect transcript, the bridge from spoken form to tool argument is locale-specific, and it is usually written once against the first language and then reused.

• Spoken numerals do not map uniformly. Digit grouping, the decimal separator, and language-specific magnitudes — a lakh, a man, a milliard — all require per-locale parsing. A normaliser that assumes thousands and millions will silently produce a figure two orders of magnitude off.
• Dates are ambiguous in exactly the way that matters. Day-month versus month-day is a coin flip that resolves to a valid date roughly half the time, so the error never raises an exception. Resolve it from the caller's locale, not from the model's habit, and confirm anything consequential by ear.
• Addresses are structured differently, not just written differently. Field order, postal code shape, and what counts as a required component all vary. A single address parser tuned to one country degrades quietly on the others.
• Currency and units need a source of truth outside the transcript. "Two thousand" in a bilingual call needs a currency from the account, not from the language of the sentence.
• Normalise once, at the boundary, and log both forms. Keep the raw recognised text alongside the normalised argument in the trace. When a tool call turns out to have been wrong, the only way to tell a recognition failure from a normalisation failure is to have both, and traces that keep only the parsed value make that diagnosis impossible.

Evaluate on real bilingual audio, and report per language.

Every failure above is invisible to the test set most teams build, because that test set is synthesised, monolingual, and clean. Fixing the evaluation is what turns this page into an actionable programme.

• Record the golden set from real speakers, including bilingual ones. Text-to-speech test audio does not code-switch naturally, does not carry a bilingual speaker's phonetics, and will pass a stack that fails in production. Recorded audio is the only input that exercises the layer that breaks.
• Report every metric per language, and never only in aggregate. A pooled success rate dominated by your largest language will stay flat while a smaller one collapses. Language is a routing key; treat it as a dimension on every dashboard, the way per-customer economics treats tenants.
• Add a code-switched slice as its own suite. Not a few examples inside the main set — a named slice with its own threshold, because it is the slice that regresses independently of the others when a vendor updates a model.
• Score the language decision separately from the transcript. Log what the detector chose, on what evidence, and whether it revised. Detection accuracy is a first-class metric and it is the one that predicts the others.
• Set an escalation path per language before launch. If the agent cannot reliably serve a language, routing to a human who speaks it is a legitimate answer and a far better one than serving it badly. Decide it deliberately — the handoff design is in interruption and handoff.
• Re-baseline on every vendor model update. Recognition quality per language moves independently between releases, and a change that improves your headline language can regress a secondary one. This is the multilingual instance of quality regression detection.

Related: STT, TTS and speech-to-speech for the components underneath, realtime agent architecture for where the routing decision lives, multilingual and cross-lingual agents for the retrieval half of the same problem, and telephony and PSTN integration for why the 8 kHz path makes all of this harder.