The second phase is the benchmark, and nobody has reported it yet

At a glance

Two artifacts from the same group: a benchmark published in September 2026, and an agent harness released in October that runs it.

Phase A: what 0.229 actually tells you

The task is harder than the framing suggests

The agent is handed a CWE description and read-only access to a repository, and nothing else: no advisory text, no file hints, no stack trace. It must return the set of files the vulnerability lives in, scored as file-set F1 against ground truth. That setup is deliberately close to the real job of a triage engineer who has a scanner alert and a monorepo, and it is much harder than the "AI finds vulnerabilities" genre implies, because finding a bug is not the task — finding the bug, in the right files, is.

The published result is that the best of 27 language models and four static-analysis tools, run under the same prompt, the same read-only tools and the same command budget, reaches 0.229 File F1. And the number that should travel further than the leaderboard position: on 38.4% of the 500 tasks, no evaluated model finds a single correct file. Not a low score — a zero, nearly four times in ten.

The six-thousandths gap is the interesting measurement

Second place is Antares-3B, a purpose-trained specialist, at 0.223. A three-billion-parameter model finishing six thousandths of a point behind the best general frontier system on a repository-scale reasoning task is a statement about the task, not about either model: whatever is missing here is not parameters. The honest reading is that localization at repository scale is bounded by something else — search strategy, context budget, the fact that the evidence is spread across files no single retrieval pass returns — which is the same bound the search-strategy literature keeps running into.

Caveat worth stating plainly: these figures come from the group's own announcements and technical report, not from an independent evaluation, and the Antares-3B weights are not released (the 350M and 1B models are open-weight). A number a vendor reports about its own second-place model on its own benchmark is not fabricated, but it is also not replicated, and the distinction matters when the gap being claimed is 0.006.

Phase B: the capacity to punish a false positive

What the second phase does

Phase B hands the agent the post-fix repository — the same codebase with the patch applied — and scores it on true-negative rate: did it correctly report that there is no vulnerability of this class here? That one addition changes what the benchmark is measuring. Phase A alone is maximised by a strategy with no judgement in it at all: return every plausible file, accept the precision penalty, bank the recall. F1 does punish that, but it punishes it inside a single number from which you cannot recover which half the agent is bad at. Phase B externalises the second half, and it does it on an input where the right answer is silence.

Silence is the output almost no agent benchmark scores, and it is the output a production triage tool spends most of its time producing. A scanner that runs on every pull request sees a vulnerability on a tiny minority of them; the thing that determines whether anybody keeps the integration turned on is what it does on the overwhelming majority where the answer is "nothing here." That is the number Phase B defines.

And it has not been reported

Here is the gap. The Antares technical report states that Phase B is included in the benchmark specification but is not evaluated in that work. So the design exists, the harness to run it exists as of October, and the figure that would let you compare agents on their false-positive behaviour is not yet public for any system on the list. Every ranking currently in circulation is a Phase A ranking.

This is not a criticism of the group that built it — specifying the phase and saying out loud that you have not run it is considerably better practice than quietly omitting it. It is a criticism of how the numbers will be used. A 0.229 on Phase A will be cited as the state of the art in agentic vulnerability analysis, and the measurement that would tell you whether any of those systems can be deployed without drowning a security team in noise does not exist yet.

A benchmark with no true-negative capacity cannot distinguish a careful agent from an indiscriminate one. Adding the capacity to the specification is half the work; the other half is running it.

The harness is making an argument too

Agentic FAITH, released in October, is the part that will matter longest, because a benchmark is only as reusable as the thing that runs it. Three of its choices are worth copying into your own eval rig.

• Read-only repository access, enforced by the environment. Not a prompt instruction — a property of the sandbox. An agent that could edit the repository could make Phase B trivially pass, and the only way to rule that out is to make the write impossible rather than discouraged. This is the eval-integrity question in its simplest form.
• Two execution targets. Docker locally for trusted inputs, Kubernetes via Agent Sandbox for everything else. Running a security benchmark means executing code from 290 real repositories, which is the sandboxing problem and not an afterthought.
• Checkpointed tasks. Execution records are saved and completed tasks checkpointed, so an interrupted run reuses eligible results instead of starting over. Dull, and it is the difference between a 500-task benchmark you re-run after every change and one you run twice a year — see the cost of evaluation.

The same command budget applied to every system is the fourth choice and the one most private evals get wrong. Without it, a comparison between an agent allowed 40 tool calls and one allowed 400 is a comparison of budgets; budget-matched evaluation is the general form of the problem.

What to take into your own eval set

You are probably not benchmarking vulnerability localization. The transferable part is structural, and it applies to any agent whose job is to notice something.

Every detector eval needs a negative half. If your eval set contains only cases where something is wrong, your scores are a measure of sensitivity with no companion measure of restraint, and the agent that improves fastest against it will be the one that learns to always say yes. Build the clean cases deliberately: the same input with the problem removed, scored on correct silence. Report the two numbers separately, because a blended score is a trade-off you cannot read back out.

Make "nothing found" a first-class output. It needs a schema, a confidence, and somewhere in the trace to record what was checked — otherwise "no finding" and "did not look" are the same string in your logs. Evaluating guardrails & detectors covers the asymmetry in detail, and calibration is the property you are actually trying to measure.

Expect zeros, and report them. A mean F1 of 0.229 is a much friendlier number than "no model solved 38.4% of tasks," and the second one is the one that predicts your Monday. Report the share of tasks where every system scored zero; it is the cheapest honesty available in benchmark reporting.

Further reading

On this wiki:

• Evaluating Guardrails & Detectors — the negative half, generalised.
• Eval Integrity & Scorer Gaming — why read-only has to be enforced, not requested.
• Budget-Matched Evaluation — the same command budget for every system.
• Vulnerability Remediation Agents — what happens downstream of a localization that is right.
• Reading Benchmarks Critically — the habit this post is an instance of.

Sources:

• VLoc Bench — benchmark page, task counts and leaderboard.
• Introducing Agentic Evaluation in FAITH — the harness, the two phases and the execution targets.
• Antares technical report — the 0.223 figure and the statement that Phase B is not evaluated.