A research agent that invents a citation fails completely, and the most-quoted benchmark in this category cannot see it happen — BrowseComp grades one short string, so a fabricated source is not a wrong answer, it is invisible. These four benchmarks disagree about what the answer is, and that disagreement, not the ranking, is what decides which failure each one can detect. Pick the one that scores the failure you cannot afford.
At a glance
Four benchmarks, four different graded objects. The dates and counts come from each project's own paper or documentation.
| Benchmark | Released | Public items | What it grades |
|---|---|---|---|
| BrowseComp | OpenAI, 10 Apr 2025 | 1,266 questions | One short, verifiable string |
| Gaia2 | Meta, 2025, on ARE | ~800 validation scenarios | Seven capabilities, equally weighted |
| Mind2Web 2 | OSU-NLP, NeurIPS 2025 | 130 tasks (10 public dev) | The answer and its attribution |
| DeepResearch Bench | ICLR 2026 | 100 tasks, 22 fields | Report quality and citations, separately |
BrowseComp — a short string, and the honesty of a hard ceiling
OpenAI released BrowseComp on 10 April 2025: 1,266 hard, fact-seeking questions, trimmed from an original 1,287 after 21 were removed for bad or ambiguous labels. Each question has one short answer that is difficult to find and easy to verify, and accuracy is the share graded correct by a semantic-equivalence checker. The design is deliberately narrow and it buys two real things — a large item count and a grader nobody can argue with.
The headline numbers, and why the gap is the point
In the paper, GPT-4o with browsing scored 1.9% and o1 scored 9.9%, while Deep Research scored 51.5%. A spread that wide on the same items is genuinely informative: it says the capability being measured is scaffold-and-training-dependent rather than a smooth function of model quality. The paper is also candid that Deep Research was trained specifically on BrowseComp-like tasks, so 51.5% is not a clean out-of-distribution measurement — the benchmark contamination problem in its mildest and most honest form, disclosed by the authors rather than discovered by an auditor.
The number in the same paper that nobody quotes
BrowseComp also measured calibration, and the result is far more useful than the accuracy figure. Deep Research showed a 91% calibration error; GPT-4o with browsing showed 82% with browsing against 69% without. In other words, the browsing agent was more confident and less right about its own confidence than the same model answering from memory. The pass-rate analysis makes the same point from another angle: across 64 trials per question, Deep Research solved 16% of tasks every time and 14% never — so roughly 70% of the set is a coin-weighting exercise where a single run tells you little.
If you deploy on one run and report accuracy, you are reporting a draw from that 70%. Eval variance and statistical power is the arithmetic; the practical form is that a 2-point difference between two agents on a single BrowseComp pass is noise.
Gaia2 — grading the trajectory instead of the answer
Gaia2 is Meta's successor to GAIA, and it changes the object under test. Where GAIA asked whether an agent could find an answer across three difficulty levels, Gaia2 scores seven agent capabilities with equal weight in the final number — execution, search, adaptability, temporal reasoning, ambiguity handling and multi-agent collaboration among them. It ships on Meta's open Agents Research Environments (ARE) framework, so the environment is runnable and debuggable rather than a static question list, with roughly 800 public validation scenarios across five configurations and a 160-scenario Gaia2-mini subset. The test set stays private, and scoring combines a model judge (Llama 3.3 Instruct 70B) with exact match depending on the task.
Equal weighting is the design decision to argue with, and it is a feature for one purpose and a bug for another. Averaging seven capabilities gives you a single comparable number across agents — useful for tracking a frontier. It also means an agent that is excellent at five and hopeless at temporal reasoning lands near an agent with the inverse profile, and those two agents are not interchangeable in any product. Read the per-capability breakdown, not the composite; the same warning applies to every aggregate in reading agent benchmarks.
What Gaia2 does not grade is attribution. A trajectory-level benchmark asks whether the agent took sensible actions and reached the right state, which is the right question for an assistant acting in an environment and the wrong one for a deliverable a human will cite. Trajectory and process evaluation covers when this framing is the one you want.
Mind2Web 2 — the only one that grades the sources
Mind2Web 2 (OSU-NLP Group, NeurIPS 2025 Datasets and Benchmarks track) is the smallest set here and the most expensive to build: 130 long-horizon, time-varying web tasks constructed with over 1,000 hours of human labour, split into a 10-task public development set and a 120-task private test set used for the leaderboard.
Agent-as-a-Judge, and why 130 tasks is enough
The grading is the contribution. Rather than one judge prompt, the authors build task-specific judge agents from tree-structured rubrics, and each task carries dozens to hundreds of fine-grained evaluation nodes. That is why a 130-task benchmark still discriminates: the unit of measurement is a rubric node, not a task, so the effective sample is much larger than the item count suggests. A human evaluation of the judges reported 99.03% correctness on automated leaf-node judgments.
Critically, the rubrics check both answer correctness and source attribution. This is the only benchmark of the four where "the agent asserted a true fact and attributed it to a page that does not say so" is a scored failure rather than a pass. If you are shipping research output that someone will act on, that is the single most important property in this post. The paper reports the strongest system reaching roughly half to 70% of human performance while using about half the time — a useful shape, because it says the remaining gap is quality rather than throughput.
DeepResearch Bench — two scores, deliberately not combined
DeepResearch Bench (ICLR 2026) grades a written report, which is what a deep-research product actually emits. It is 100 PhD-level research tasks, 50 in Chinese and 50 in English, spanning 22 fields including science and technology, finance and business, software engineering and art and design — the only one of the four with first-class non-English coverage, which matters if your users are not all writing in English.
It scores on two axes that the authors keep apart. RACE scores report quality — comprehensiveness, depth, instruction-following and readability, combined as a weighted sum. FACT scores retrieval and citation behaviour through citation accuracy and average effective citations. And the finding that justifies the separation: the two are only weakly correlated at the agent level. A high RACE does not predict a high FACT.
That is the same observation as BrowseComp's calibration result, arrived at from the opposite end. BrowseComp found that being right and knowing you are right come apart; DeepResearch Bench found that writing well and citing honestly come apart. Both say a single number cannot stand in for research quality, because the dimensions are empirically independent — and the dimension that is dropped when you collapse them is always the one about truthfulness.
If you only have budget for one axis, instrument FACT-shaped citation checking rather than RACE-shaped quality scoring. Quality failures are visible to any reader; attribution failures are not, and they are the ones that void the deliverable. Citations and source attribution UX is the product half of the same point.
Cross-cutting: three ways these benchmarks diverge
What counts as a hallucination
On BrowseComp, a hallucination is a wrong string and nothing else — the grader never looks at a source, so an unsupported-but-correct answer and a well-sourced correct answer score identically. On Gaia2, a hallucination shows up only if it derails the trajectory or corrupts the final state. On Mind2Web 2 and DeepResearch Bench, an unsupported claim is a scored defect in its own right. Four benchmarks, three different definitions of the core failure mode in hallucination and grounding.
How fast the ground truth rots
BrowseComp's answers are chosen to be stable facts, which is what lets a frozen 1,266-item set stay valid. Mind2Web 2's tasks are explicitly time-varying — live prices, current listings, present-day availability — which makes them a far better proxy for production and far more expensive to maintain, because a task whose correct answer changed last Tuesday is now a mislabelled item. Gaia2 sidesteps this by making the environment itself the ground truth. DeepResearch Bench sits in between: the tasks are durable but FACT's citation checks run against a live web.
Who holds the private split
Three of the four keep a private test set — Gaia2's is reserved for internal validation, Mind2Web 2 holds back 120 of 130 tasks, and only BrowseComp is fully public. That is the right call for leaderboard integrity and it has a cost you should price in: you cannot run the real evaluation yourself, which means you cannot run it on your own agent, your own scaffold or your own model. For internal work you are reproducing the method on your own items, not running the benchmark. Eval integrity and scorer gaming is why they do it; budget-matched evaluation is what you have to control for when you do.
When to pick which
| Your situation | Use | Because |
|---|---|---|
| Tracking raw retrieval capability across models | BrowseComp | Large public set, unarguable grader, no attribution noise |
| Building an assistant that acts in an environment | Gaia2 | Runnable environments, per-capability breakdown, adaptability scored |
| Shipping research output a human will cite | Mind2Web 2 | The only one that scores whether a source actually supports the claim |
| Shipping long-form reports, or non-English | DeepResearch Bench | Grades the report as the artefact, and keeps citations on their own axis |
And for anything internal: take the method rather than the score. Mind2Web 2's tree-structured rubrics and DeepResearch Bench's two-axis split are both reproducible on fifty of your own tasks in a week, and they measure your product. A public number measures somebody else's scaffold on somebody else's items — which is the standing argument of the benchmark landscape.
FAQ
Which benchmark should I quote for a deep-research agent?
None of them alone. BrowseComp for retrieval reach, and either Mind2Web 2 or DeepResearch Bench for whether the output is attributable. Quoting an accuracy figure without an attribution figure describes half the product.
Why is BrowseComp's calibration result more useful than its accuracy?
Because accuracy tells you how often the agent is right and calibration tells you whether you can believe it when it says it is. The paper reported a 91% calibration error for Deep Research, meaning confident wrong answers were common — which is the failure that reaches a user.
Is a 130-task benchmark big enough to trust?
Mind2Web 2 is, because each task carries dozens to hundreds of graded rubric nodes, so the effective sample is far larger than the task count. A 130-task benchmark graded with one pass/fail per task would not be.
Can I run these on my own agent?
Only partly. BrowseComp is fully public; Gaia2 and Mind2Web 2 keep private test splits, so you can develop against the public portion but the leaderboard score is not something you can reproduce locally. For internal evaluation, copy the grading method onto your own tasks.
Does a high report-quality score imply good citations?
No. DeepResearch Bench found RACE and FACT only weakly correlated at the agent level, so report quality and citation trustworthiness have to be measured and reported separately.
Further reading
On this wiki:
- Reading agent benchmarks — how to read any of these numbers without being fooled by them.
- The benchmark landscape — where these four sit among everything else.
- Research and synthesis agents — why a fabricated source voids the whole deliverable.
- Measuring calibration in production — the BrowseComp calibration finding, applied to your own traffic.
- Uncertainty and calibration — the concept, in plain terms.