Deep-Dives / Reasoning & Test-Time Compute
Reasoning & Test-Time Compute
Chain of thought, self-consistency, tree/graph of thought, and the inference-time-scaling laws that govern them.
- Chain-of-Thought, ProperlyFaithfulness and monitorability are different questions with different answers, and merging them is why teams either file the trace as an audit log or delete it. It is still a poor explanation. But OpenAI's CoT-Control (March 2026, 13 reasoning models) measured suppression instead: Claude Sonnet 4.5 controlled its chain of thought 2.7% of the time against 61.9% for its final output, and controllability falls with longer thinking, more RL and harder problems — every lever that buys accuracy. So the trace is a weak explanation and a high-recall sensor you did not design, structured output is the channel models control at 61.9% and therefore not your signal, and trace visibility is a procurement question before it is a monitoring one.
- Self-Consistency & SamplingWhy sampling + majority vote works, the exact bias-amplification failure, the saturating returns curve, and how to spend the k budget.
- Tree & Graph of ThoughtDeliberate search over partial solutions, the multiplicative cost, and the load-bearing dependency on a partial-state scorer.
- Verifier-Guided SearchOutcome vs process reward models steering best-of-N and beam search, reward hacking at inference time, and why the verifier is the product.
- Inference-Time ScalingTest-time compute as a second scaling axis, the difficulty-adaptive compute-optimal frontier, and where more thinking stops paying.
- When Reasoning Helps (and When It Burns Money)The synthesis decision rule — task class × verifiability × budget — an escalation ladder, the named money-burning patterns, and a do/don't list.
- Adaptive Thinking & Effort Budgets`budget_tokens` is deprecated — Claude's `effort`, Gemini's `thinking_level`, OpenAI's `reasoning_effort`, and when the model overrides your budget.
- Carrying Reasoning Across Tool CallsReasoning became a signed, opaque input you must hand back unchanged — so injecting a reminder, adding a tool mid-run or trimming history now invalidates it, silently or with a 400. Echo, never rebuild.
- Latent Reasoning & the Trace You Stop GettingContinuous thought and recurrent depth buy real capability — a two-layer transformer with D continuous steps solves graph reachability where discrete CoT needs O(n²) decoding steps, and a 3.5B recurrent model reaches ~50B-equivalent compute at 50 loops — by deleting the artifact five production controls consume. Name those five, concede that the trace was never faithful and keep it anyway because monitoring it still beats monitoring actions alone, then fit probes on the early latent steps where the signal concentrates — which makes this a procurement question, because a hosted latent model gives you no residual stream to read.
- Inverse Scaling in Test-Time ComputeTurning the thinking budget up is the knob everyone trusts, and on four identifiable task shapes accuracy falls as reasoning length rises — distractor-heavy extraction, estimation where the obvious feature is wrong, constraint-tracking deduction, and framing overfit. The failure splits by model family, so effort is part of the model identity rather than a tuning parameter, and an eval run at one effort level cannot see the regression the next upgrade ships.
- Accounting for Reasoning TokensReasoning tokens are the only line on your invoice with no matching object in your logs, and the three major providers report them in incompatible shapes — nested inside the output count for OpenAI and Anthropic, a sibling of it for Gemini — so the one piece of arithmetic every cross-vendor cost dashboard copies overstates one vendor by a multiple and understates the other by the same. Normalise once at the edge keyed on (vendor, model, surface), default a missing field to null rather than zero, reconcile per model against the invoice rather than per provider, and remember that continuity is billed twice: once at the output rate and then at the input rate on every turn the blob rides along in.
- Summaries & Sealed Blobs: the Reasoning Your API ReturnsYou cannot store the reasoning a model produced — only one of two substitutes with opposite properties: a readable summary that is a fresh generation about the reasoning, and a sealed blob the provider treats as faithful and you cannot read. The sealed one is a protocol obligation (Anthropic rejects a request whose thinking blocks were modified; a missing Gemini thought_signature is a 400 on function calling), and it is byte-fragile — one OpenClaw report traced persistent rejections to a Unicode sanitiser rewriting signed text. Faithfulness audits, process scoring and incident receipts built on the readable artifact are measuring the summariser.
- Budget Awareness, Not Budget SizeThinking longer and calling more tools plateau for the same reason — the model has no representation of its remaining allowance, so extra budget widens whatever it was already doing. One 2025 study added a budget counter to the context and lifted resolve rate from 36.8% to 55.4% at a similar tool-call count, which is why a plateau under more budget is an allocation problem, not a resourcing one: reserve 10-20% for landing, split terminal reasons into finished/gave up/truncated, and raise a cap only when truncations rise.