Deep-Dives / Reasoning & Test-Time Compute

Reasoning & Test-Time Compute

Chain of thought, self-consistency, tree/graph of thought, and the inference-time-scaling laws that govern them.

  1. Chain-of-Thought, Properly
    Faithfulness and monitorability are different questions with different answers, and merging them is why teams either file the trace as an audit log or delete it. It is still a poor explanation. But OpenAI's CoT-Control (March 2026, 13 reasoning models) measured suppression instead: Claude Sonnet 4.5 controlled its chain of thought 2.7% of the time against 61.9% for its final output, and controllability falls with longer thinking, more RL and harder problems — every lever that buys accuracy. So the trace is a weak explanation and a high-recall sensor you did not design, structured output is the channel models control at 61.9% and therefore not your signal, and trace visibility is a procurement question before it is a monitoring one.
  2. Self-Consistency & Sampling
    Why sampling + majority vote works, the exact bias-amplification failure, the saturating returns curve, and how to spend the k budget.
  3. Tree & Graph of Thought
    Deliberate search over partial solutions, the multiplicative cost, and the load-bearing dependency on a partial-state scorer.
  4. Verifier-Guided Search
    Outcome vs process reward models steering best-of-N and beam search, reward hacking at inference time, and why the verifier is the product.
  5. Inference-Time Scaling
    Test-time compute as a second scaling axis, the difficulty-adaptive compute-optimal frontier, and where more thinking stops paying.
  6. When Reasoning Helps (and When It Burns Money)
    The synthesis decision rule — task class × verifiability × budget — an escalation ladder, the named money-burning patterns, and a do/don't list.
  7. Adaptive Thinking & Effort Budgets
    `budget_tokens` is deprecated — Claude's `effort`, Gemini's `thinking_level`, OpenAI's `reasoning_effort`, and when the model overrides your budget.
  8. Carrying Reasoning Across Tool Calls
    Reasoning became a signed, opaque input you must hand back unchanged — so injecting a reminder, adding a tool mid-run or trimming history now invalidates it, silently or with a 400. Echo, never rebuild.
  9. Latent Reasoning & the Trace You Stop Getting
    Continuous thought and recurrent depth buy real capability — a two-layer transformer with D continuous steps solves graph reachability where discrete CoT needs O(n²) decoding steps, and a 3.5B recurrent model reaches ~50B-equivalent compute at 50 loops — by deleting the artifact five production controls consume. Name those five, concede that the trace was never faithful and keep it anyway because monitoring it still beats monitoring actions alone, then fit probes on the early latent steps where the signal concentrates — which makes this a procurement question, because a hosted latent model gives you no residual stream to read.
  10. Inverse Scaling in Test-Time Compute
    Turning the thinking budget up is the knob everyone trusts, and on four identifiable task shapes accuracy falls as reasoning length rises — distractor-heavy extraction, estimation where the obvious feature is wrong, constraint-tracking deduction, and framing overfit. The failure splits by model family, so effort is part of the model identity rather than a tuning parameter, and an eval run at one effort level cannot see the regression the next upgrade ships.
  11. Accounting for Reasoning Tokens
    Reasoning tokens are the only line on your invoice with no matching object in your logs, and the three major providers report them in incompatible shapes — nested inside the output count for OpenAI and Anthropic, a sibling of it for Gemini — so the one piece of arithmetic every cross-vendor cost dashboard copies overstates one vendor by a multiple and understates the other by the same. Normalise once at the edge keyed on (vendor, model, surface), default a missing field to null rather than zero, reconcile per model against the invoice rather than per provider, and remember that continuity is billed twice: once at the output rate and then at the input rate on every turn the blob rides along in.
  12. Summaries & Sealed Blobs: the Reasoning Your API Returns
    You cannot store the reasoning a model produced — only one of two substitutes with opposite properties: a readable summary that is a fresh generation about the reasoning, and a sealed blob the provider treats as faithful and you cannot read. The sealed one is a protocol obligation (Anthropic rejects a request whose thinking blocks were modified; a missing Gemini thought_signature is a 400 on function calling), and it is byte-fragile — one OpenClaw report traced persistent rejections to a Unicode sanitiser rewriting signed text. Faithfulness audits, process scoring and incident receipts built on the readable artifact are measuring the summariser.
  13. Budget Awareness, Not Budget Size
    Thinking longer and calling more tools plateau for the same reason — the model has no representation of its remaining allowance, so extra budget widens whatever it was already doing. One 2025 study added a budget counter to the context and lifted resolve rate from 36.8% to 55.4% at a similar tool-call count, which is why a plateau under more budget is an allocation problem, not a resourcing one: reserve 10-20% for landing, split terminal reasons into finished/gave up/truncated, and raise a cap only when truncations rise.