Multilingual & cross-lingual agents.
A system that works in English does not degrade gracefully in another language — it breaks in three separate places at once, and generation quality, the part everyone tests, is usually the least broken of them. The same meaning costs roughly twice the tokens in Chinese, your embedding model may not connect a Chinese question to an English document at all, and an English-only eval set can see none of it happening.
The token tax is real and it compounds.
Tokenizers are fitted to their training corpus, and that corpus is overwhelmingly English. English averages roughly four characters per token; Chinese lands closer to one or two, so the same content becomes about twice as many tokens. Languages written in scripts the tokenizer barely saw fare worse — three to four times is common for several Indic and Southeast Asian scripts, where characters can fall apart into individual bytes.
- Your bill doubles for identical work. Same users, same questions, twice the tokens in and out.
- Your context window halves. A 200K window holds meaningfully less meaning in Chinese, which squeezes retrieved documents and few-shot examples against each other — see context windows.
- Latency rises with output length. More output tokens means more decode steps, and decode is the sequential half of the request; see prefill and decode.
- Truncation arrives earlier. Limits tuned against English inputs start cutting real content, usually silently.
The tempting inversion — write prompts in the cheaper language to save money — mostly does not work. Measured on coding tasks, switching the prompt language away from the model's strongest one gives back in solve rate roughly what it saves in tokens, and sometimes more. Choose language for accuracy; treat the token cost as a budgeting fact, not a lever.
Retrieval is where it actually breaks.
This is the failure that a fluent answer hides. An embedding model trained on English places a Chinese query nowhere near the English document that answers it, so the retriever does not return a slightly worse passage — it returns an irrelevant one, and the model writes a confident answer from it.
- Multilingual embedding model. One index, queries in any language matching documents in any language. Models like BGE-M3 and Qwen3-Embedding are built for this. Costs a little quality against a best-in-class monolingual model, and buys enormous simplicity.
- Translate at index time. Store each document in every supported language. Doubles or triples the index, and bakes translation errors in permanently — every future query inherits them.
- Translate the query. Cheap, fast, one extra call. Loses named entities, product names, and idioms exactly where precision matters most.
- Keyword search fails quietly. Hybrid retrieval leans on BM25, and BM25 needs word boundaries. Chinese, Japanese, and Thai have no spaces, so without a language-aware analyser your keyword half tokenizes into nonsense and contributes nothing — while still returning results. See chunking and vector search.
- Chunk boundaries shift too. A fixed character count is a different amount of content per language, so your carefully-tuned chunk size is no longer what you tuned.
The instruction and safety layers leak.
Everything wrapped around the model was written in one language, and that language exerts a pull.
- Language drift. The model answers in the language of the strongest nearby signal — often your English system prompt, or an English tool result that arrived mid-conversation. Pin the output language explicitly and test it as a requirement, not a preference.
- Structured outputs need the rule spelled out. Schema keys stay English; values must be localised. Say which is which in the field descriptions or you will get a mix — see structured outputs.
- Safety classifiers are usually English-tuned. A moderation or policy check that never fires on non-English input is worse than no check, because it reports green. The same holds for guardrails built on keyword lists.
- Injection crosses languages before your filter does. An instruction hidden in a document in a third language routes around English pattern-matching while the model reads it perfectly well — one more reason prompt injection defences cannot be lexical.
Evaluate per language or you are guessing.
Every failure above is invisible in an aggregate score. If a tenth of your traffic is in a second language and half of it fails, your headline metric moves five points and looks like noise.
- Segment every metric by language. Not one multilingual score — one score per language, on the same tasks, tracked separately. See evals.
- Do not translate the eval set with the model under test. It will translate in exactly the way it misreads, and the test set will agree with the bug. Use a fluent human, or at minimum a different model.
- Score retrieval separately from the answer. Cross-lingual retrieval failure produces fluent, well-formed, wrong output, which end-to-end scoring frequently marks as merely mediocre rather than broken.
- Have a native speaker read the register. Formality, honorifics, measure words, and sentence rhythm are what make output read as translated. Facts can be right while the text still costs you the reader's trust.
Do the cheap diagnosis first. Take twenty real queries in your second language, run them through retrieval only, and read what comes back. In systems built in English this is where the failure almost always is, and it is undetectable from the final answer, because the model will write something plausible from whatever documents it was handed. Fix the retriever before you touch a single prompt.
Related: tokens & tokenization for why the tax exists, RAG for the pipeline this sits inside, and small & local models for the multilingual embedding options.