BrowseComp vs Gaia2 vs Mind2Web 2 vs DeepResearch Bench

The four benchmarks for web-research agents disagree about what the answer even is — a short string, a trajectory, a claim plus its sources, or a written report — and that choice, not the leaderboard, fixes which failure each one can see. BrowseComp reported 51.5% accuracy for Deep Research alongside a 91% calibration error in the same paper, and DeepResearch Bench found report quality and citation trustworthiness only weakly correlated at the agent level. Pick by the failure you cannot afford.

By Agentic AI Wiki 11 min read

A research agent that invents a citation fails completely, and the most-quoted benchmark in this category cannot see it happen — BrowseComp grades one short string, so a fabricated source is not a wrong answer, it is invisible. These four benchmarks disagree about what the answer is, and that disagreement, not the ranking, is what decides which failure each one can detect. Pick the one that scores the failure you cannot afford.

At a glance

Four benchmarks, four different graded objects. The dates and counts come from each project's own paper or documentation.

BenchmarkReleasedPublic itemsWhat it grades
BrowseComp OpenAI, 10 Apr 2025 1,266 questions One short, verifiable string
Gaia2 Meta, 2025, on ARE ~800 validation scenarios Seven capabilities, equally weighted
Mind2Web 2 OSU-NLP, NeurIPS 2025 130 tasks (10 public dev) The answer and its attribution
DeepResearch Bench ICLR 2026 100 tasks, 22 fields Report quality and citations, separately
Public task count per web-research benchmark A horizontal bar chart of publicly available items. BrowseComp has 1,266 questions, Gaia2 about 800 public validation scenarios, Mind2Web 2 has 130 tasks of which 10 are a public development set, and DeepResearch Bench has 100 tasks. Grading depth runs the opposite way: the two smallest sets grade dozens to hundreds of rubric nodes per task, while the largest grades one short string. Public items per benchmark BrowseComp 1,266 Gaia2 ~800 Mind2Web 2 130 DeepResearch Bench 100 325 650 975 1,300 Grading depth runs the other way: the two smallest sets score dozens to hundreds of rubric nodes per task.
Item count and grading depth run in opposite directions, and that is the trade, not an accident.
What each web-research benchmark treats as the answer A four by three matrix. BrowseComp: the answer is a short string, graded by a semantic-equivalence checker, and a fabricated citation is invisible to it. Gaia2: the answer is a trajectory across seven capabilities, graded by a model judge plus exact match, and attribution is out of scope. Mind2Web 2: the answer is a claim plus its sources, graded by task-specific judge agents over tree-structured rubrics, and attribution is scored directly. DeepResearch Bench: the answer is a written report, graded by RACE for quality and FACT for citations, and attribution is a separate score. Legend: attribution scored, partly in scope, not visible. What counts as the answer The graded object Who grades it Sees a fake citation? BrowseComp A short string Equivalence checker No Gaia2 A trajectory Judge + exact match Out of scope Mind2Web 2 A claim and its sources Rubric judge agents Yes, scored directly DeepResearch Bench A written report RACE and FACT Yes, a separate score Attribution scored Partly in scope Not visible
Only two of the four score attribution at all. That column is the whole argument.

BrowseComp — a short string, and the honesty of a hard ceiling

OpenAI released BrowseComp on 10 April 2025: 1,266 hard, fact-seeking questions, trimmed from an original 1,287 after 21 were removed for bad or ambiguous labels. Each question has one short answer that is difficult to find and easy to verify, and accuracy is the share graded correct by a semantic-equivalence checker. The design is deliberately narrow and it buys two real things — a large item count and a grader nobody can argue with.

The headline numbers, and why the gap is the point

In the paper, GPT-4o with browsing scored 1.9% and o1 scored 9.9%, while Deep Research scored 51.5%. A spread that wide on the same items is genuinely informative: it says the capability being measured is scaffold-and-training-dependent rather than a smooth function of model quality. The paper is also candid that Deep Research was trained specifically on BrowseComp-like tasks, so 51.5% is not a clean out-of-distribution measurement — the benchmark contamination problem in its mildest and most honest form, disclosed by the authors rather than discovered by an auditor.

The number in the same paper that nobody quotes

BrowseComp also measured calibration, and the result is far more useful than the accuracy figure. Deep Research showed a 91% calibration error; GPT-4o with browsing showed 82% with browsing against 69% without. In other words, the browsing agent was more confident and less right about its own confidence than the same model answering from memory. The pass-rate analysis makes the same point from another angle: across 64 trials per question, Deep Research solved 16% of tasks every time and 14% never — so roughly 70% of the set is a coin-weighting exercise where a single run tells you little.

If you deploy on one run and report accuracy, you are reporting a draw from that 70%. Eval variance and statistical power is the arithmetic; the practical form is that a 2-point difference between two agents on a single BrowseComp pass is noise.

Gaia2 — grading the trajectory instead of the answer

Gaia2 is Meta's successor to GAIA, and it changes the object under test. Where GAIA asked whether an agent could find an answer across three difficulty levels, Gaia2 scores seven agent capabilities with equal weight in the final number — execution, search, adaptability, temporal reasoning, ambiguity handling and multi-agent collaboration among them. It ships on Meta's open Agents Research Environments (ARE) framework, so the environment is runnable and debuggable rather than a static question list, with roughly 800 public validation scenarios across five configurations and a 160-scenario Gaia2-mini subset. The test set stays private, and scoring combines a model judge (Llama 3.3 Instruct 70B) with exact match depending on the task.

Equal weighting is the design decision to argue with, and it is a feature for one purpose and a bug for another. Averaging seven capabilities gives you a single comparable number across agents — useful for tracking a frontier. It also means an agent that is excellent at five and hopeless at temporal reasoning lands near an agent with the inverse profile, and those two agents are not interchangeable in any product. Read the per-capability breakdown, not the composite; the same warning applies to every aggregate in reading agent benchmarks.

What Gaia2 does not grade is attribution. A trajectory-level benchmark asks whether the agent took sensible actions and reached the right state, which is the right question for an assistant acting in an environment and the wrong one for a deliverable a human will cite. Trajectory and process evaluation covers when this framing is the one you want.

Mind2Web 2 — the only one that grades the sources

Mind2Web 2 (OSU-NLP Group, NeurIPS 2025 Datasets and Benchmarks track) is the smallest set here and the most expensive to build: 130 long-horizon, time-varying web tasks constructed with over 1,000 hours of human labour, split into a 10-task public development set and a 120-task private test set used for the leaderboard.

Agent-as-a-Judge, and why 130 tasks is enough

The grading is the contribution. Rather than one judge prompt, the authors build task-specific judge agents from tree-structured rubrics, and each task carries dozens to hundreds of fine-grained evaluation nodes. That is why a 130-task benchmark still discriminates: the unit of measurement is a rubric node, not a task, so the effective sample is much larger than the item count suggests. A human evaluation of the judges reported 99.03% correctness on automated leaf-node judgments.

Critically, the rubrics check both answer correctness and source attribution. This is the only benchmark of the four where "the agent asserted a true fact and attributed it to a page that does not say so" is a scored failure rather than a pass. If you are shipping research output that someone will act on, that is the single most important property in this post. The paper reports the strongest system reaching roughly half to 70% of human performance while using about half the time — a useful shape, because it says the remaining gap is quality rather than throughput.

DeepResearch Bench — two scores, deliberately not combined

DeepResearch Bench (ICLR 2026) grades a written report, which is what a deep-research product actually emits. It is 100 PhD-level research tasks, 50 in Chinese and 50 in English, spanning 22 fields including science and technology, finance and business, software engineering and art and design — the only one of the four with first-class non-English coverage, which matters if your users are not all writing in English.

It scores on two axes that the authors keep apart. RACE scores report quality — comprehensiveness, depth, instruction-following and readability, combined as a weighted sum. FACT scores retrieval and citation behaviour through citation accuracy and average effective citations. And the finding that justifies the separation: the two are only weakly correlated at the agent level. A high RACE does not predict a high FACT.

That is the same observation as BrowseComp's calibration result, arrived at from the opposite end. BrowseComp found that being right and knowing you are right come apart; DeepResearch Bench found that writing well and citing honestly come apart. Both say a single number cannot stand in for research quality, because the dimensions are empirically independent — and the dimension that is dropped when you collapse them is always the one about truthfulness.

If you only have budget for one axis, instrument FACT-shaped citation checking rather than RACE-shaped quality scoring. Quality failures are visible to any reader; attribution failures are not, and they are the ones that void the deliverable. Citations and source attribution UX is the product half of the same point.

Cross-cutting: three ways these benchmarks diverge

What counts as a hallucination

On BrowseComp, a hallucination is a wrong string and nothing else — the grader never looks at a source, so an unsupported-but-correct answer and a well-sourced correct answer score identically. On Gaia2, a hallucination shows up only if it derails the trajectory or corrupts the final state. On Mind2Web 2 and DeepResearch Bench, an unsupported claim is a scored defect in its own right. Four benchmarks, three different definitions of the core failure mode in hallucination and grounding.

How fast the ground truth rots

BrowseComp's answers are chosen to be stable facts, which is what lets a frozen 1,266-item set stay valid. Mind2Web 2's tasks are explicitly time-varying — live prices, current listings, present-day availability — which makes them a far better proxy for production and far more expensive to maintain, because a task whose correct answer changed last Tuesday is now a mislabelled item. Gaia2 sidesteps this by making the environment itself the ground truth. DeepResearch Bench sits in between: the tasks are durable but FACT's citation checks run against a live web.

Who holds the private split

Three of the four keep a private test set — Gaia2's is reserved for internal validation, Mind2Web 2 holds back 120 of 130 tasks, and only BrowseComp is fully public. That is the right call for leaderboard integrity and it has a cost you should price in: you cannot run the real evaluation yourself, which means you cannot run it on your own agent, your own scaffold or your own model. For internal work you are reproducing the method on your own items, not running the benchmark. Eval integrity and scorer gaming is why they do it; budget-matched evaluation is what you have to control for when you do.

When to pick which

Your situationUseBecause
Tracking raw retrieval capability across models BrowseComp Large public set, unarguable grader, no attribution noise
Building an assistant that acts in an environment Gaia2 Runnable environments, per-capability breakdown, adaptability scored
Shipping research output a human will cite Mind2Web 2 The only one that scores whether a source actually supports the claim
Shipping long-form reports, or non-English DeepResearch Bench Grades the report as the artefact, and keeps citations on their own axis

And for anything internal: take the method rather than the score. Mind2Web 2's tree-structured rubrics and DeepResearch Bench's two-axis split are both reproducible on fifty of your own tasks in a week, and they measure your product. A public number measures somebody else's scaffold on somebody else's items — which is the standing argument of the benchmark landscape.

FAQ

Which benchmark should I quote for a deep-research agent?

None of them alone. BrowseComp for retrieval reach, and either Mind2Web 2 or DeepResearch Bench for whether the output is attributable. Quoting an accuracy figure without an attribution figure describes half the product.

Why is BrowseComp's calibration result more useful than its accuracy?

Because accuracy tells you how often the agent is right and calibration tells you whether you can believe it when it says it is. The paper reported a 91% calibration error for Deep Research, meaning confident wrong answers were common — which is the failure that reaches a user.

Is a 130-task benchmark big enough to trust?

Mind2Web 2 is, because each task carries dozens to hundreds of graded rubric nodes, so the effective sample is far larger than the task count. A 130-task benchmark graded with one pass/fail per task would not be.

Can I run these on my own agent?

Only partly. BrowseComp is fully public; Gaia2 and Mind2Web 2 keep private test splits, so you can develop against the public portion but the leaderboard score is not something you can reproduce locally. For internal evaluation, copy the grading method onto your own tasks.

Does a high report-quality score imply good citations?

No. DeepResearch Bench found RACE and FACT only weakly correlated at the agent level, so report quality and citation trustworthiness have to be measured and reported separately.

Further reading

On this wiki:

Project sources: