Model routing & cascades.
Routing only pays when deciding "is this request hard?" is cheaper and more reliable than simply answering it — and for most open-ended work it is neither, which is why the published 85%-savings figures come from chat benchmarks and rarely survive contact with an agent. The version that does survive is unglamorous and almost risk-free: route by task in your own code, at the sub-steps you already know are mechanical, and reach for a learned router or a cascade only where you have a cheap automatic verifier.
Three mechanisms, wrongly treated as one.
- Static routing. You decide, in code, which model handles which job: the classification step, the extraction step and the reranker go to a small model; the reasoning loop goes to the frontier model. No inference-time decision, no new failure mode, and it captures most of the money that is actually on the table.
- Dynamic routing. A classifier — a small model, an embedding lookup, a learned scorer — reads each incoming request and picks a model for it. Adds a decision, a latency cost, and a component that can be wrong in a way nothing downstream detects.
- Cascading. Send everything to the cheap model first, evaluate the answer, escalate to the expensive model only when the answer fails a check. Strictly more powerful than dynamic routing, because it judges the actual output rather than predicting difficulty from the input — and strictly more dependent on that check being trustworthy.
A fourth thing gets called routing and is not: an LLM gateway that gives you one API key across many providers, with failover and spend tracking. That is plumbing, and worth having, but it does not choose a model on quality grounds. Do not let the shared vocabulary convince you that installing a gateway has done the routing work.
The escalation signal is the entire problem.
A cascade's economics are simple: it pays when verification costs much less than generation. That condition is met more rarely than teams assume, and the quality of your check — not the size of the price gap — decides whether the whole scheme saves money or quietly degrades your product.
- Genuinely cheap verifiers exist, and they are all external. Code that must compile or pass tests. Output that must validate against a schema. A retrieved-answer claim that must be traceable to a source document. A tool call whose arguments must resolve to real records. Where you have one of these, a cascade is excellent.
- Self-reported confidence is not one. Asking a model how sure it is produces a fluent number that correlates weakly with correctness, and a weak model is confidently wrong in exactly the cases you built the cascade to catch.
- A judge model is a real cost, not a free check. If the judge has to be strong enough to be right, you are paying near-strong-model prices on every request to avoid paying them on some — and on hard cases you now pay twice, once for the failed cheap attempt and once for the escalation.
Work the arithmetic before you build. With a strong model at 10× the cheap one, a cascade that escalates 30% of traffic and adds a verifier at 5% of strong-model cost lands near 40% of the all-strong bill — a real win. Push escalation to 60% and add a judge at 20%, and the saving is mostly gone while you have added latency and a component to debug. The break-even is much closer than the headline numbers suggest.
A router is a model, so it needs the treatment you give models.
Once a classifier picks models at request time, it is a component in your critical path with its own accuracy, its own drift, and its own bill — and it is invisible in a way most components are not, because a bad routing decision surfaces as a slightly worse answer rather than as an error.
- Evaluate it directly. Not "did the system answer well" but "for the requests sent to the cheap model, would the strong model have done better?" That is a labelling job on real traffic, and it is the only number that tells you whether the router is earning its place. Treat it as an eval like any other.
- It drifts as your traffic drifts. A router trained or tuned on last quarter's requests degrades silently as usage shifts. Re-measure on a schedule; it does not announce its own decay.
- Behaviour stops being uniform. Two users asking near-identical questions get different models, different formatting habits and different failure modes. For consumer chat this is tolerable; for a product where output feeds a downstream parser it is a source of bugs that reproduce only sometimes.
- Log the decision. Which model served this request, and why, belongs in the trace next to the tokens — see observability. Without it, every quality investigation starts by not knowing which model produced the output.
- Published savings are not transferable. Router benchmarks are overwhelmingly conversational. Agent work — tool-call generation, schema compliance, long-horizon consistency — degrades differently and often earlier on small models, so validate on your own tasks. This is the same caution as reading benchmarks critically.
Route the mechanical steps; leave the loop alone.
Inside an agent, the highest-volume calls are usually not the reasoning ones. Classification, extraction, reranking, summarising a tool result, deciding whether a document is relevant — these run many times per task, have checkable outputs, and are exactly where a small or local model is good enough at a fraction of the price. That is small & local models in practice, and it needs no router at all: you already know which step is which.
The reasoning loop is the opposite case. Downgrading it is the intervention that looks cheapest and usually is not, because a weaker model takes more steps, and in an agent every extra step re-sends the whole transcript — the quadratic growth that makes "just use the cheap model" disappoint. Capability that ends the task sooner is a cost lever.
- Measure per completed task, never per token. A routing change that cuts token spend 40% and success rate 15% is a loss, and only the per-task metric shows it.
- Keep one escape hatch. A way to force the strong model for a given request or tenant, so a routing regression is a config change rather than a deploy.
- Pin the routes. Each destination is a model version with its own behaviour; version them explicitly and re-run your evals when any of them moves — see rollout, versioning & pinning.
Start with static routing by task type. It is a switch statement, it carries no router risk, and in most agent systems it captures the large majority of the available savings because the mechanical sub-steps dominate call volume. Add a cascade only where a cheap automatic verifier already exists — tests, a schema, a grounding check — and add a learned router only after you have measured that static routing left money on the table. Most teams never reach that third step, and are right not to.
Related: cost, quality & latency for the underlying trade-off, choosing a model for picking each destination, and reasoning models for the other dial that changes spend per request.