Benchmark Contamination & Leakage

Three different leaks wear the same word.

"Contamination" gets used for at least three failures with different causes, different detection methods, and different fixes. Conflating them is why the conversation goes in circles.

• Verbatim leakage. The test items themselves — questions and answers — were in the pre-training corpus. This is the version everyone pictures, it is the easiest to detect, and on modern benchmarks it is increasingly the least important, because benchmark authors now check for it.
• Solution leakage. The answer was never in the benchmark file, but it was reachable from the artifacts the benchmark points at. Agent benchmarks are built out of real repositories, real issues, real ticket threads — and the fix for a GitHub issue is very often sitting in a comment on that same issue, or in the commit that closed it, or in the release notes. An audit of the original SWE-bench found roughly a third of accepted patches had the fix visible somewhere in the issue thread, and roughly another third passed only because the accompanying tests were too weak to separate a correct patch from a plausible one.
• Indirect leakage. Nobody trained on the test set, but the corpus is full of blog posts, leaderboard analyses, papers and tutorials about the benchmark: which items are hard, what the common failure patterns are, what a good answer looks like. This is undetectable by n-gram overlap, it grows with a benchmark's popularity, and it is a large part of why a benchmark's discriminative power decays in the years after release even when the items were never published in solved form.

The property belongs to the pair, not the benchmark.

Say "SWE-bench is contaminated" and you have said something almost meaningless. The same benchmark is clean with respect to a model trained before it was published and dirty with respect to one trained after; a model whose lab ran aggressive decontamination and one that scraped GitHub indiscriminately can sit on the same leaderboard with wildly different exposure to the same items. What you actually have is a matrix of (model, benchmark) cells, each one time-stamped.

That reframing changes what you can conclude:

• Cross-model comparison is the operation contamination breaks first. Ranking is the thing everyone uses a benchmark for and the thing contamination destroys soonest, because two models can differ in exposure by more than they differ in ability. A model released a year after the benchmark carries an advantage that has nothing to do with being better.
• Absolute scores degrade more gracefully than ranks. A contaminated score is still an upper bound on some real capability — the model did produce a passing patch. Treat public numbers as a floor on "can it do this class of task at all", never as a measure of the margin between two candidates.
• The decay is monotone and roughly predictable. A benchmark's useful life starts the day it is published and ends somewhere after the first model generation trained on a corpus that includes it, plus its commentary. Two years is a reasonable default assumption for a popular public set; you should be actively suspicious of any leaderboard whose items are older than the models on it.

This is the mechanism behind the saturation described in the 2026 benchmark landscape. When the top five entries sit within a point of each other, contamination differential alone is comfortably large enough to produce that ordering, and no amount of statistical care on your side can separate the signal — a point the variance and statistical power discussion reaches from the other direction.

Four tests you can run without the training data.

You will never get to inspect a frontier lab's corpus, and detection methods that need it are academic. These four work from the outside, cost little, and are ordered by how much they tell you per unit of effort.

• The date split. Build a small set of tasks from material created strictly after the model's knowledge cutoff — issues filed last month, papers published last quarter — and compare performance against structurally matched older tasks. A large gap is the cleanest contamination signal available, and it needs no clever statistics. It is also the single most useful thing you can build, because the same set doubles as your ongoing regression suite.
• Perturbation sensitivity. Rename the variables, reorder the multiple-choice options, rewrite the issue text while preserving the problem. A model reasoning about the task should be nearly indifferent; a model retrieving a memorized answer degrades sharply. Do not over-read a small drop — genuine brittleness produces the same signature — but a collapse is diagnostic.
• Memorization probes. Ask the model to reproduce the item rather than solve it: complete the issue text from its first line, list the files in the repository at that commit, state the test names. A model that can recite the environment has seen the environment. This is the agent-specific probe and it catches things a solution-overlap check will not.
• Public-vs-held-out delta. If the benchmark ships a public split and a private one, the gap between them is a direct read on exposure. Where no private split exists, the same logic applies to the difference between the published leaderboard number and the number you reproduce on your own harness — though be careful, because harness differences alone routinely move agent scores by several points.

What contamination looks like in an agent benchmark specifically.

Agent evaluation makes the problem worse in a way that single-turn QA does not, and the reason is that an agent benchmark ships an environment, not just a question.

A coding agent scored on a popular open-source repository is being tested on a codebase that is, with near certainty, in its training data in full. It knows the directory layout, the naming conventions, where the tests live, which module owns which concern. Every one of those is a real advantage over the same agent dropped into your private monorepo, and none of it is measured. The score you are reading is not "can this agent navigate an unfamiliar codebase"; it is "can this agent navigate a codebase it has already read", and the first is the capability you are buying.

• Navigation contamination is invisible to answer-overlap checks. The patch can be entirely novel while the search that found the right file was free. This is why the memorization probe in step 3 — can it list the files? — is more informative here than any comparison of solutions.
• Tool and API surfaces carry the same problem. An agent benchmarked against a well-documented public API is exercising memorized call signatures. Point it at your internal service, where the only documentation is your own tool schemas, and the gap is often startling.
• Trajectory scoring inherits it too. If you grade the path as well as the outcome, a memorized path scores as good process. See trajectory and process evaluation for what the path is supposed to be telling you.

The uncomfortable implication is that the headline agentic-coding numbers systematically overstate performance on private code, and that the size of the overstatement is specific to how much your codebase resembles the public ones. That is not a reason to ignore the benchmarks. It is a reason to expect a drop when you run your own, and to treat the size of that drop as information about your environment rather than as a defect in the model.

The designs that buy time, and what each one costs.

The field's response has converged on four strategies. None of them solves the problem; each trades a different cost for a longer half-life.

• Continuously refreshed benchmarks. Harvest new tasks automatically from live sources — issues filed this month, papers posted this week — and retire old ones. This is the only approach that genuinely outruns indirect leakage, and its cost is that scores are no longer comparable across time: this quarter's items are not last quarter's items, so a rising score may be an easier draw. Report the sampling procedure or the number means nothing.
• Private held-out splits. Keep half the items unpublished and run submissions yourself. Effective and expensive, and it converts the benchmark into a service with an operator, a queue, and a trust requirement — you are now taking someone's word for a number you cannot reproduce.
• Contamination-resistant construction. Generate items procedurally, or from private data, or in a form where the answer cannot be stated in text. Buys the most time; usually buys it by moving away from the realistic tasks that made the benchmark worth having.
• Human curation of existing sets. The "Verified" pattern: keep the raw material, pay people to remove items whose solution is visible and whose tests do not discriminate. Does nothing about verbatim or indirect leakage, but it is the highest-return fix for solution leakage and it makes the remaining number mean what it says.

For an eval you own, the same menu applies at a smaller scale, and the calculus is different because you control the source. Your date-split set from step 3 is a continuously refreshed benchmark; your production traffic is private held-out data that no lab can have trained on. That is the real advantage of an in-house suite, and it is worth more than its size suggests — see eval-driven development and CI.

The decision rule.

Everything above collapses into one operating principle: public benchmarks answer "can it, at all"; only data the model could not have seen answers "which one, and by how much". Sort your questions into those two buckets and the contamination problem mostly stops being a problem, because you have stopped asking public numbers to do the job they cannot do.

• Screening a shortlist — public benchmarks are fine and cheap. Use them to eliminate candidates that clearly cannot do the task class. Do not use them to order the survivors.
• Choosing between two finalists — public benchmarks are worthless, and the difference between them is almost certainly smaller than the contamination differential. Run your own set. See choosing a model.
• Tracking your own progress over time — contamination is not the threat here, dataset rot is: your set gets easier as you fix the things it caught. Add tasks from recent failures continuously.
• Reporting a number to someone else — date it. "82% on tasks harvested in July 2026" is an honest claim that survives scrutiny; "82% on <benchmark>" is a claim whose meaning changes under the reader without either of you noticing.

Related: reading agent benchmarks critically for the harness-sensitivity half of the same scepticism, HAL and asynchronous agent eval for what a benchmark measures beyond accuracy, and evals 101 for the foundations.