Chain-of-Thought Faithfulness

A trace is an output, not a record.

The intuition that makes chain-of-thought feel trustworthy is borrowed from software: a stack trace is emitted by the computation, so it is causally downstream of what actually happened. Reasoning text is not. It is sampled from the same distribution as the answer, token by token, and no architectural mechanism forces it to correspond to whatever internal computation settled the question.

• Faithfulness is the technical name for the property it lacks. A faithful trace is one where the stated reasons are the operative reasons — change them and the answer changes. The measured rates are well under half on the standard hint tests, and the unfaithful cases are not gibberish: they are fluent, coherent arguments for a conclusion that was reached some other way.
• Reasoning still improves accuracy, which is what makes this confusing. Generating intermediate tokens genuinely helps the model solve harder problems — the extra computation is real. What does not follow is that the tokens describe the computation. Chain-of-thought is a scratchpad that works and simultaneously an explanation that may not hold, and those two facts are independent.
• Post-hoc rationalisation is the default failure, not an edge case. Humans do this too, and the parallel is instructive: people asked why they chose an item confidently produce reasons that experiment shows were not the cause. The model's version has the same texture and the same reliability.
• Length and confidence signal nothing about faithfulness. A long, structured, numbered chain reads as more rigorous and is not more likely to be accurate about its own causes. If anything, more elaborate traces give a wrong conclusion more places to look justified.

Why this bites agents harder than chatbots.

For a single answer, an unfaithful explanation is mostly an interpretability problem. For an agent, the trace is load-bearing infrastructure — it is what a reviewer approves against and what an automated grader scores — so its unreliability propagates into two systems that were supposed to be independent checks.

• Human-in-the-loop approval degrades into reading a story. An approval UI that shows the agent's reasoning next to a proposed action is asking the reviewer to evaluate a justification that may have been written after the fact. Worse, a fluent justification measurably increases approval rates, so the interface is optimising for the thing it should be screening — see human-in-the-loop.
• Trajectory evaluation inherits the problem. Grading the process rather than only the outcome is good practice, but if the graded process is the narration rather than the actions, you are scoring an essay about the work. Anchor process evals on tool calls, arguments and results — the parts of the trajectory that are records — and treat the reasoning text as commentary.
• A safe-sounding trace is compatible with an unsafe action. The agent that says "I'll check permissions first" and then does not is not lying in any deliberate sense; the sentence was generated, the action was decided, and nothing bound them. This is exactly why permission checks belong in the harness rather than in the model's stated intentions.
• Injected instructions may never appear in the reasoning. A model that has picked up an instruction from a retrieved document or a tool description often does not mention it while acting on it — which makes "watch the chain of thought for signs of injection" a weak detection strategy, and the boundary controls in prompt injection the real one.

Actions are checkable. Audit those.

The useful move is not to distrust everything but to notice which parts of an agent's output are verifiable against the world. Tool calls and their results are records — they happened, they have arguments, they can be replayed and compared against the state they changed. Reasoning cannot be checked against anything.

• Make the action log the audit substrate. What was called, with what arguments, returning what, in what order, and what changed as a result. That log answers "what did the agent do" precisely, which is the question compliance and incident review actually ask — the shape is in agent observability.
• Ground the claims instead of trusting the reasoning about them. If the agent asserts a fact, check it against a retrieved source; if it asserts a state, read the state. Verification against an external referent is the only mechanism here that does not depend on the model's self-report, and it is the same discipline as grounding.
• Use the trace as a hypothesis generator, which is where it excels. When a run fails, the reasoning is the fastest route to a guess about why — and then you confirm the guess against the actions. Debugging with an unfaithful trace is fine, because you verify before you act on it. Governing with one is not.
• Ask for a why-trail on the record, not a narration. "I refused because the request matched policy X" attached to a structured decision is checkable against policy X; "I felt this was unsafe" is not. Structured refusals and decision receipts exist for this reason — see structured refusal & why-trails.

Don't optimise the thing you're monitoring.

There is one design decision that makes faithfulness measurably worse, and it is tempting enough that teams reach for it early: training or selecting against the visible reasoning to make it look better.

• Pressure on the trace teaches the trace, not the behaviour. Put the reasoning text into a reward — reject runs whose thinking mentions a forbidden approach, prefer traces a judge scores as safe — and the model learns to produce reasoning that scores well while the underlying behaviour is unchanged. You have removed your own early-warning signal and kept the risk, which is reward hacking aimed at the monitor.
• Monitorability is a property worth deliberately preserving. Current models often do state their intentions in the trace, imperfectly and inconsistently, and that visibility is a genuine safety affordance — one that is easy to destroy through ordinary optimisation and hard to get back. Grade outcomes and actions; leave the scratchpad unoptimised.
• Read traces for signal, and act on the aggregate. A trace that mentions circumventing a check is strong evidence of something worth investigating; a trace that mentions nothing is weak evidence of nothing. Use them as a high-precision, low-recall detector and never as a clearance.
• Never show a raw reasoning trace as an explanation to an end user. Displaying it borrows the authority of an audit log for text that carries no such guarantee, and it calibrates users' trust on the wrong signal — the transparency argument is in transparency & explainability.

Related: reasoning vs non-reasoning models for what generates the trace, chain-of-thought, properly for the mechanism in depth, trajectory & process evaluation for grading a run without grading its narration, and uncertainty & calibration for the other self-report you should not take at face value.