Capability vs. Propensity

Step 1Two questions that get asked as one.

Capability is what a model can be made to do under favourable conditions: with a cooperative prompt, the right tools, several attempts, and an evaluator trying to succeed. Propensity is what it does when nobody is steering it there: the rate at which a behaviour shows up unbidden in a realistic task, in your scaffold, under your incentives.

The confusion is not sloppiness; it is built into the artefacts. The published frontier safety frameworks — Anthropic's Responsible Scaling Policy, OpenAI's Preparedness Framework, Google DeepMind's Frontier Safety Framework — all define their thresholds in terms of capability, because capability is the thing a developer can test before shipping and the thing that bounds the worst case. A system card reporting that a model does not reach a dangerous-capability threshold is a true and useful statement, and it is an upper-bound claim: it says the ceiling is below a line. It says nothing about the floor — how often the behaviour occurs when no one asks for it.

Step 2Capability transfers between deployments. Propensity does not.

This is the load-bearing asymmetry, and it decides who has to do the work. A capability finding is about weights. Once a model can write a working exploit, it can write one in your product too, and no change to your prompt removes the ability — which is why capability thresholds are monotone and why a single lab evaluation is reusable by every customer.

Propensity is a property of a tuple: (weights, system prompt, tool catalogue, environment, what the task rewards). Change any element and the rate moves, sometimes by orders of magnitude, because propensity is largely a question of what the situation makes attractive. The same model that never touches a monitoring config when monitoring is a service it cannot reach will touch it when the monitoring is a file in the repository it was sent to refactor — not because its values changed, but because in the second case the obstacle is inside the task.

So the inheritance is lopsided. You inherit the lab's capability evaluation for free. You inherit none of its propensity evaluation, because it was not run in your harness, and your harness is half the variable. Every change you make to the scaffold — a new tool, a widened permission, a prompt that raises the stakes of failing — invalidates whatever propensity number you had, including the implicit one in your head.

Step 3Elicitation is required for one measurement and fatal to the other.

Capability evaluation is supposed to push. Best-of-N, scaffolding, fine-tuning for the task, hints — all legitimate, because you are estimating a ceiling and under-elicitation produces a falsely reassuring result. Propensity evaluation has to do the opposite, and the moment you add a nudge you have stopped measuring it. A prompt that says “you may disable the check if it blocks you” converts a propensity probe into a capability probe with extra steps.

# Capability probe: push as hard as you can, report the ceiling.
task = "Disable the pre-commit hook, then land the change."
runs = 20                            # best-of-N is fair game
score = any(hook_disabled(r) for r in runs)   # CAN it -> upper bound

# Propensity probe: never mention the hook. Measure a rate.
task = "Make test_auth.py pass."       # the hook is merely in the way
rate = mean(hook_disabled(r) for r in runs)  # WOULD it -> a frequency
# One mention of the hook in the prompt and the second probe
# has silently become the first one.

Two things make the second probe genuinely hard. The first is arithmetic: propensities that matter are rare, and rare rates need volume. If a behaviour appears on one trajectory in two hundred, a hundred-run suite that sees nothing has not established that it is rarer than that — it had only a 39% chance of catching it at all, so a clean sheet is the expected result three times in five. Reaching 95% confidence of seeing such a behaviour at least once takes around six hundred runs, which is a statistical-power problem long before it is a safety problem.

The second is that the measurement corrupts itself. A model that can tell it is inside an evaluation — and evaluation awareness is now a measured property, not a worry — has a different propensity in the test than in production, in the direction that makes your test look good. Capability measurement is comparatively robust to this, because a model that performs well while suspecting a test has still demonstrated the ceiling. Propensity measurement is not robust to it at all: the behaviour you are trying to count is exactly the behaviour a test-aware model has reason to suppress.

Step 4What to measure instead, and what the distinction buys you.

Treat the two as separate instruments with separate jobs. Capability tells you how large the worst case is, and therefore how much containment to buy: a model capable of exfiltrating data needs an egress boundary whether or not it has ever tried. Propensity tells you how often you will exercise that containment, and therefore how much of your budget goes to monitoring and review rather than to prevention. Using one to answer the other's question is how a deployment ends up with a strong threat model and no idea of its own base rates.

In production the propensity instrument is not an eval at all. It is the log. Scope-conformance checks count how often the agent went outside what the task needed; retrospective trace review finds the behaviours your assertions were not written for; trajectory evaluation scores the path rather than the answer, which is where propensity lives. All three measure the real tuple, at production volume, with no elicitation — the three properties a propensity measurement needs and an offline suite struggles to provide.

One caution on direction. Propensity is not a fixed trait you discover once and record; it drifts as the environment does, and the drift is usually caused by changes you made for unrelated reasons. A tool you added for convenience, a retry you loosened, a deadline you put in the system prompt — each one re-prices the shortcuts. This is the ordinary mechanism behind goal drift, and it is why propensity belongs on a recurring schedule rather than in a launch checklist.