Graceful Degradation & Fallback

Swapping the model swaps the agent.

For a stateless completion, a fallback is roughly interchangeable: worse prose, same shape. For an agent, the model is the component that decides which tools to call, when to stop, and whether to refuse — so replacing it replaces the control logic while every downstream assumption stays where it was.

• Tool-calling behaviour is the first thing to break, and it breaks quietly. Models differ in schema dialect, in how strictly they honour required fields, in whether they emit parallel calls, and in how many tools they can hold before selection accuracy collapses. A twelve-tool agent that is reliable on the primary can start guessing arguments on the fallback, and the symptom is not an exception — it is a plausible wrong action.
• Context ceilings differ, so the fallback may not be able to hold the task at all. A run that compacts at 200k tokens and fails over to a model with a smaller effective window does not degrade; it truncates. Whatever fell out of the window is invisible to every log you have — the distinction between advertised and usable context is in long context: effective vs advertised.
• Refusal and safety profiles are not aligned across vendors. A fallback that refuses a category your primary handles turns an outage into a support queue; one that accepts a category your primary refuses turns an outage into an incident. Neither shows up in a latency dashboard.
• Structured output is where it usually surfaces first. Constrained decoding support varies by provider and by model tier, and a fallback without it will return JSON that is right 96% of the time — which is fine for a demo and an outage for a pipeline that parses it.
• The step count changes, and with it the cost. A weaker model takes more turns to reach the same place, and every turn re-sends a growing transcript. A cheaper fallback is routinely more expensive per completed task, which is the arithmetic in agent cost control pointed at your worst hour.

Decide what to shed before you decide what to swap.

Model failover is the reflex, and it is rarely the highest-return control. Most agent wall-clock and most agent failures come from the tool layer, and most of the load you need to remove in a degraded hour is work the task did not strictly require.

• Write a ranked shed list, in advance, per task type. Which enrichment steps, which optional retrievals, which verification passes, which reranking, which self-critique. When capacity is short, drop from the bottom in a fixed order. A list decided in a design review is worth ten times a list improvised during an incident.
• Reasoning effort is a dial before the model is a switch. Cutting thinking budget, dropping self-consistency from five samples to one, or turning off the reflection pass keeps the same model, the same tool dialect and the same refusal profile while returning a large share of the capacity. Reach for it first — the trade-off is mapped in adaptive thinking & effort budgets.
• Shed by tenant and by task class, not uniformly. Uniform degradation spends your remaining capacity on batch jobs that could have waited and on free-tier traffic, while the interactive request someone is watching gets the same haircut. Priority classes decided in advance are the difference between a degraded hour and a lost one.
• Queue rather than degrade when the work is not interactive. A nightly batch does not need a weaker model, it needs to run at 04:00 instead of 02:00. Deferring is the cheapest form of degradation available and it costs no quality at all.
• Caches are load-bearing in exactly the wrong direction. A semantic cache raises its hit rate under degradation because traffic concentrates, which is helpful — but the entries it serves were generated by the primary, so your degraded output is a blend of two models with no marker saying which produced what. Stamp cache entries with the configuration that produced them.

Fail open on reads. Fail closed on anything with a side effect.

The direction of failure is a per-component decision, and getting it backwards in either direction is how a degradation becomes an incident. The dividing line is whether the component's absence changes what the agent does or only what it knows.

• A retrieval backend that is down should degrade to "answering without sources", explicitly and visibly. The agent must say it, the trace must record it, and any downstream claim-checking must know it. Silently answering from parametric memory when the index is unavailable is the highest-severity failure on this page, because the output looks identical and is no longer grounded.
• A guardrail or policy service that is down must fail closed. If the injection classifier, the PII filter or the authorization check cannot be reached, the action does not proceed. A safety control that fails open is a safety control that is strongest exactly when it is least needed. This is the same argument as the in-loop kill switch.
• A write tool that is degraded must never be retried optimistically. Degradation multiplies retries, and retries multiply side effects. Every write path needs an intent-derived idempotency key before you build any of this, per idempotency, retries and side-effect safety; without it, degraded mode is the hour you send the same email four times.
• Lower the autonomy threshold when quality drops. Auto-approve rules were calibrated against the primary's accuracy. If the degraded configuration scores materially lower on your evals, the actions it may take unsupervised must shrink to match — a degraded agent with unchanged permissions is the worst combination on offer, and it is the default one.
• Never fail over silently across a compliance boundary. A fallback in a different region or on a different provider can move data somewhere your contracts do not allow. Region and vendor are part of the fallback's identity, and that check belongs in configuration rather than in someone's memory — see data residency & sovereignty.

Degraded mode is a named state with an exit, or it becomes permanent.

The most common ending for a well-built fallback is not a failure. It is that the system entered degraded mode during an incident, the incident resolved, nobody noticed, and three weeks later the product is quietly running on the cheap model with the reduced tool set.

• Make it an explicit state, not an emergent property of retry logic. mode=degraded on the run, on every span, on every log line and on a dashboard. If the only way to know you are degraded is to infer it from error rates, you will not know.
• Alert on entry, on duration, and on the fraction of traffic affected. Entry alone is noise. "12% of runs degraded for the last 40 minutes" is an incident; "one run degraded" is a Tuesday.
• Recovery needs an explicit health check and hysteresis. Return to primary when it has been healthy for a defined window, not on the first successful probe — otherwise a flapping provider produces a system that oscillates between two agents with different behaviour, which is worse for users than staying degraded.
• Give every degradation an owner and an expiry. A flag that turns off the verification pass needs a date on it. Flags without expiries are how a temporary trade-off becomes architecture; the discipline is in feature flags for agents.
• Tell the user, in the product, in one line. "Running in reduced mode — sources unavailable" costs nothing and preserves the trust calibration the whole interface depends on. Users forgive a degraded answer they were warned about and do not forgive a confident wrong one they were not.

Exercise it, because an untested fallback does not exist.

The degraded path has near-zero production hours by construction. Every other part of your system is tested continuously by traffic; this one is tested only by outages, which is the worst possible time to discover that the fallback's API key expired in March.

• Route a small share of real traffic through the fallback continuously. One or two percent, permanently. It keeps credentials valid, keeps the code path compiled and exercised, and — the real prize — produces an ongoing quality comparison against the primary on live traffic rather than on a stale eval set.
• Run a scheduled degradation drill. Force the primary to fail for fifteen minutes in production, on a schedule, with the team watching. The first drill always finds something: an unset timeout, a fallback that inherits a prompt referencing a tool it does not have, a retry storm, an alert that never fires.
• Test the transitions, not just the states. Most degradation bugs live in the switch — a run that fails over mid-trajectory and loses its tool-call history, or a cached prefix that no longer matches the new model and silently costs full price. Failing over between steps of a live run is a distinct test case from starting a run degraded.
• Include the exit in the drill. Recovery is the half nobody rehearses, and hysteresis bugs only appear when the primary comes back.
• Keep the fallback's eval results current. Re-run the suite against the degraded configuration on the same cadence as the primary. A fallback qualified nine months ago against a model version that has since been retired is a configuration nobody has evaluated at all — which is also the failure mode in model deprecation & migration.

Five numbers that say whether degradation is working.

Availability alone will report success through an hour in which every answer was ungrounded. These say something more useful.

• Time in degraded mode, and share of runs affected. The exposure metric. Trend it weekly; a slow rise means the primary's capacity planning is drifting, not that the fallback is working well.
• Quality delta, primary versus degraded, on the same eval suite. The number that turns degradation into a decision. If you cannot state it, you do not know what your outage costs — and you cannot set the autonomy threshold for degraded mode.
• Task completion rate in degraded mode. Not response rate. A degraded agent that returns something for every request while completing half the tasks is a system reporting 100% availability and delivering 50% of the product.
• Silent-degradation incidents. Count of times you discovered after the fact that you had been degraded without an alert firing. The target is zero and the first measurement is usually not.
• Cost per completed task, degraded versus primary. Frequently higher, always surprising, and the fastest way to end the assumption that the cheap fallback is saving money during an outage.

Related: rate limits & provider capacity for admission control upstream of any of this, SLOs & error budgets for what degradation spends, model routing & cascades for the same swap made deliberately, and designing for failure for what the user sees.