Reasoning vs non-reasoning models.
"Reasoning model" stopped being a thing you choose and became a budget you spend, and the budget is spent before you can see it: thinking tokens are billed at the output rate on every major vendor, counted inside the same output cap as the answer, and invisible until the bill arrives. So the design question is no longer which model to buy — it is which steps of your loop are allowed to think, and the answer is far fewer than the default setting assumes.
Step 1What "reasoning" means mechanically, and what it costs.
A non-reasoning ("direct") model reads the prompt and immediately starts emitting the answer. A reasoning model first generates an internal chain of intermediate tokens — exploring approaches, doing sub-steps, checking itself — and only then emits the visible answer. That intermediate stream is variously called thinking, reasoning, or scratchpad tokens. It is usually hidden or summarized in the API, but it is real generation, and three consequences follow that decide every cost conversation you will have about it:
- Thinking is billed at the output rate. On Anthropic, OpenAI and Google alike, reasoning tokens cost what completion tokens cost, whether or not you are shown them. There is no discount for being hidden.
- Thinking shares the output cap with the answer. Your
max_tokensceiling covers both, which produces the single most common first-contact bug: a request with a tight output cap returns an empty response, because the budget was consumed thinking and nothing was left to say. Anthropic's extended thinking historically required the thinking budget to be strictly belowmax_tokens, with a floor around 1,024 tokens, for exactly this reason. - The spend is not knowable in advance. The model decides how long to think, within whatever ceiling you set. You can bound it; you cannot predict it per request, which is why reasoning cost behaves like a variable on your invoice and a constant in your pricing model.
The core idea is inference-time compute: spend more computation per question at answer time rather than more parameters or more training. It is conceptually the same effect as prompting an ordinary model to "think step by step," but trained in and far more capable — the model has learned to produce long, productive traces and to use them, instead of relying on a prompt trick.
Step 2The category collapsed into a dial, and the dial has three vocabularies.
Early on, providers shipped distinct "reasoning" families separate from their "chat" models and you picked one. As of October 2026 that line is gone: one model thinks a lot, a little, or not at all, controlled by a parameter on the request. The mental model shifts from "pick the reasoning model" to "decide how much this request is allowed to think."
Every vendor agrees on the idea and none agrees on the spelling, and the 2024–2025 examples you will find in tutorials are stale:
- Anthropic moved from an explicit token budget to adaptive effort.
budget_tokenswas deprecated on the 4.6 generation and is rejected outright on 4.7 and later; the replacement isthinking.type: adaptivewitheffort: low | medium | high. - OpenAI exposes
reasoning_effort, with a minimal rung below low. - Google uses a thinking budget on Gemini, latterly expressed as
thinking_level.
Two gotchas that bite in production. First, the model may overrule your budget upward when its own signal says it needs more, so an effort setting is a hint with a ceiling rather than a hard cap — adaptive thinking and effort budgets works through the per-vendor override behaviour. Second, the three vendors report reasoning usage in three incompatible shapes, so the obvious cross-vendor cost formula is wrong in opposite directions depending on where you point it; reasoning token accounting is the normalisation step.
Step 3More thinking is not monotonically better, and the exceptions are agent-shaped.
The intuition that higher effort is slower, dearer and safer is wrong often enough to matter. There are identifiable task families where accuracy falls as reasoning length rises — and they are not exotic. They are a prompt stuffed with irrelevant retrieved text, a judgement where the obvious feature is the misleading one, and a long list of constraints to hold simultaneously. Those are the shapes agent work actually takes, which is why effort: high is a per-task-shape decision rather than a sane default. Inverse scaling in test-time compute is the full treatment; published work has also found extended thinking failing to reliably improve accuracy on plain arithmetic.
The honest summary of when the dial pays:
- Pays. Multi-step math, logic and planning, complex code generation and review — anywhere the model benefits from exploring and rejecting options before committing, and where a wrong answer is expensive.
- Wastes. Factual lookup, classification, extraction, routing, short conversational replies. There is nothing to reason about; the extra tokens are pure cost and latency.
- Hurts. The inverse-scaling families above, plus anything where a calibrated direct answer beats a second-guessed one.
The visible trace is not a faithful account of how the answer was reached. It improves accuracy and it is a useful debugging signal; treating it as an audit log of the model's real process is a known mistake — see chain-of-thought faithfulness. And the trace may not be there at all in future: latent reasoning buys efficiency by not writing the deliberation down.
Step 4In an agent loop, the dial is per step — and here is the arithmetic.
Nothing above matters until you put it in a loop, where the multiplier is the step count. Take a twenty-step agent task and suppose, conservatively, that medium effort adds 800 thinking tokens per step over a direct call. Leave the dial up for the whole loop and you have bought 16,000 billed output tokens per task. Gate it to the two steps that genuinely plan — the initial decomposition and the one recovery step after a tool error — and you have bought 1,600. Same model, same task, a 90% reduction in the reasoning line item, and on most workloads no measurable accuracy loss, because the eighteen gated steps were tool selection and result threading, which is dispatch rather than deliberation.
That is the whole practical content of the concept. Match the budget to the step:
- Routing and tool selection: minimal or off. You want fast, predictable dispatch, and parallel tool calls matter more here than depth.
- Decomposition and recovery after failure: generous. This is where it pays for itself.
- Final synthesis: moderate or off, depending on whether the synthesis is itself hard.
Two loop-level mechanics that are easy to get wrong and expensive to discover. Reasoning now has to be handed back: on current models the reasoning block is a signed, opaque input you are required to return unchanged across a tool call, so ordinary middleware that rebuilds the message list silently deletes it or draws a 400 — carrying reasoning across tool calls is the one to read before you debug it. And because thinking tokens occupy the context window as well as the invoice, a high-effort loop eats the context budget you were saving for retrieved documents.
The default worth adopting: effort off everywhere, raised one step at a time against your own eval set, and measured as accuracy per dollar rather than accuracy. The marker that a step is wasting thinking is visible in the trace — it paraphrases the question or rambles instead of doing sub-work. Then do the thing almost nobody does: re-run that eval at the next model upgrade, because an effort policy tuned on one generation is not portable, and a regression from inverse scaling is invisible to an evaluation that only ever ran at one effort level.
Related: when reasoning helps for the full decision procedure, cost, quality and latency for the three-way trade this sits inside, agent cost control for where reasoning sits among the other levers, and choosing a model for the per-task checklist.