Two of these four do not own a verifier at all, which is the fact that makes their scores incomparable and the fact no comparison mentions. WindowsAgentArena vendors OSWorld's metric library wholesale; BrowserGym is a harness that imports whatever the wrapped benchmark wrote. So the axis that decides whether a number transfers to your product is not the operating system or the task count — it is what the check reads. Pick the benchmark whose verifier reads the same thing your definition of done reads, and ignore the rest of the comparison.
At a glance
Four projects that all get described as "GUI agent benchmarks", read against who wrote the check.
| Project | Licence | Default suite | Who owns the verifier |
|---|---|---|---|
| OSWorld (xlang-ai) | Apache-2.0 | 368 tasks, 10 domains | Itself — 225 metric functions |
| WindowsAgentArena (Microsoft) | MIT | 154 tasks, 12 domains | OSWorld — the library is vendored |
| AndroidWorld (google-research) | Apache-2.0 | 116 hand-crafted tasks, 20 apps | Itself — one success function per task |
| BrowserGym (ServiceNow) | Apache-2.0 | 8 benchmarks, incl. WebArena's 812 | Nobody — it imports each one's |
Read the last column twice. It is the whole comparison. Three of these are environments, one is a harness, and of the three environments only two wrote their own grader — because the second one copied the first.
The question that sorts them
Ask of any agent benchmark: when the run ends, what does the grader look at? Everything else follows from that. The four answers here are genuinely different kinds of thing — a file's bytes, a device's state, a web page plus a typed answer, and in one case a bag of normalised tokens scored for partial credit — and no amount of shared vocabulary ("success rate") makes them commensurable.
This is not a complaint about benchmark design. Each choice is right for what its authors were measuring. The problem is downstream: a leaderboard column headed "success rate" invites you to read 38% on one and 41% on another as a comparison, and it is not one, in the same way that a unit test pass rate and a customer-satisfaction score are both percentages.
The test that actually decides which to use: write down, in one sentence, how your product knows a task succeeded. If the answer is "a row changed in a database", you want a state-reading verifier. If it is "the user got the right answer", you want an answer-scoring one. Then pick the benchmark that already does that, and treat the other three as evidence about a different question.
OSWorld — the verifier is a library, and it acts
The mechanism
A task is a JSON file with three parts: a config that sets up the machine, an instruction in natural language, and an evaluator naming a function plus its expected and actual inputs. desktop_env/evaluators/metrics/ holds 225 functions across fifteen modules — libreoffice.py, chrome.py, pdf.py, vlc.py, vscode.py and the rest. Across the task files, the most-used verifier is compare_table (56 tasks), then infeasible (26), compare_pptx_files (23) and check_include_exclude (15).
The part teams miss
The evaluator is not a read. It has a postconfig that operates the desktop first. Here is a real one, from a LibreOffice Calc task that asks the agent to add a maturity-date column:
activate_window window_name: "MaturityDate.xlsx - LibreOffice Calc", strict: true
sleep 0.5
execute python -c 'import pyautogui; pyautogui.hotkey("ctrl","s")'
sleep 0.5
func compare_table
expected cloud_file # fetched from a HuggingFace dataset repo
result vm_file /home/user/MaturityDate.xlsx
Four things must go right before the agent's work is looked at: the window title must match exactly, no save dialog may intercept the keystroke, half a second must be enough for the write to land, and the gold file must download. A correct agent fails the task on any one of them. Across the suite the grading path issues execute 358 times, sleep 307 times, download 293 times and activate_window 200 times against 368 tasks — which puts a floor on the benchmark's own reliability that no published score reports. The general version of this problem is in when the check has to act.
The honest bits
test_infeasible.json carries 25 tasks whose correct answer is a refusal, which is a genuinely good design decision and one most suites skip. And OSWorld-Verified, announced in July 2025, fixed community-reported issues, added AWS parallelisation and re-ran model results — with the project explicitly telling you to compare against the new numbers rather than the old ones. Treat any OSWorld figure without a version as unidentified.
WindowsAgentArena — the same grader, a different machine
The mechanism
Microsoft's project is OSWorld ported to Windows 11, and it does not hide this. src/win-arena-container/client/desktop_env/evaluators/metrics/ is OSWorld's module list with edge.py added where chrome.py sits. The task-file vocabulary is the same: across its examples the funcs are exact_match (92), compare_table (48), infeasible (26), check_json_settings (20). test_all.json holds 154 tasks across 12 domains.
The part teams miss
There are two sets of task files, not one: examples/ and examples_noctxt/, 308 files for 154 tasks. That is the difficulty switch, added in November 2024 — set diff_lvl="hard" and the agent must find and open the right program itself rather than having the task config set the scene. This is the single most important thing to state alongside a WindowsAgentArena number, and it is a start_client.sh variable rather than a suite name, so it travels badly.
The cost shape
The reset unit is a 30 GB Windows 11 golden image you build yourself from an evaluation ISO, hosting a Python server that receives agent commands. The upside is that Azure ML parallelisation turns hundreds of tasks into minutes rather than days. The downside is that your results are a property of the image you built, and two teams' images are not the same artifact.
AndroidWorld — the only one where no two runs share a task
The mechanism
A task is a class with a template string, a JSON schema for its parameters, a generate_random_params() classmethod, and is_successful(env) -> float. The goal string is the template formatted with the params, so 116 hand-crafted tasks across 20 apps instantiate into what the project calls millions of unique variations. Success is read programmatically from device state after the run — 57 is_successful implementations across 96 task-eval modules. MiniWoB++ rides along inside the same harness.
The part teams miss
Randomised instantiation is the strongest anti-memorisation property in this group and the weakest reproducibility property, and it is the same design choice. A score is an average over instances nobody else drew, so two papers' AndroidWorld numbers are samples from the same distribution rather than results on the same test, and the variance that implies is not usually reported. That is a statistical-power question before it is a capability one.
The quiet parameter
Step budgets were recalibrated in November 2024 to roughly twice the human average completion time per task. That is a defensible choice, well documented, and it is also a dial: an agent that would have finished in 2.5× human time scores zero, and a suite tuned to a different multiple would rank systems differently. The budget belongs next to the score, the way it does in budget-matched evaluation.
BrowserGym — a harness, sold and read as a benchmark
The mechanism
Every task implements AbstractBrowserTask: setup(page), validate(page, chat_messages) -> (reward, done, message, info), an optional cheat(), and teardown(). Behind that interface sit eight benchmarks — MiniWoB (105 task classes), WebArena (812), WebArena-Verified (812), WebArena-Lite (165), VisualWebArena (910), WorkArena, AssistantBench and WebLINX — each installed as its own package.
The part teams miss
BrowserGym does not grade. Its WebArena task imports webarena.evaluation_harness.evaluators.evaluator_router from the upstream package and calls it; the package also pulls NLTK's punkt_tab tokenizer at import time, because WebArena's own scoring does text matching. So a "BrowserGym result" inherits every property of the benchmark underneath, including its scorer's quirks, and the uniform reward: float hides how different those are:
# same interface, incompatible semantics
WebArena task -> reward from the upstream evaluator_router
(page state + answer matching, essentially pass/fail)
AssistantBench -> reward from a token-bag F1 over normalised spans
(articles stripped, punctuation stripped, numbers
normalised, then per-span F1 aligned greedily)
-> 0.63 is a real value here, and meaningless above
Averaging across BrowserGym benchmarks is therefore averaging a pass rate with a partial-credit score. It is a reasonable thing for a harness to permit and an unreasonable thing to put in a headline.
What it is actually good for
One agent implementation against eight benchmarks, with the observation space (screenshot, accessibility tree, set-of-marks) configurable in one place. If you are building a web agent, that portability is worth more than any individual suite, and it is the right default. Just report per-benchmark numbers, never a BrowserGym average.
Task counts, and why the biggest bar is the least informative
The temptation with a chart like this is to read task count as coverage. It is closer to a reset-cost budget. A 910-task web suite runs against re-provisioned web applications; a 154-task Windows suite runs against a 30 GB image; a 116-task Android suite runs against an emulator and draws fresh parameters each time. Those are different orders of infrastructure per task, which is why the big suites are web suites and will stay that way.
The number worth asking for instead is how many distinct verifiers produced the score, because that is what bounds how much a single grader bug can move it. OSWorld's 368 tasks resolve to a few dozen functions, with compare_table alone covering 56 of them — so a regression in one table comparator moves 15% of the suite.
When to pick which
| If your product… | Pick | Because |
|---|---|---|
| Produces documents and files | OSWorld (Verified) | Its verifiers compare artifacts, which is your definition of done. |
| Automates Windows line-of-business apps | WindowsAgentArena, hard | Only suite on real Windows; the hard mode removes the task-config scaffolding. |
| Drives mobile apps, or needs anti-memorisation | AndroidWorld | Randomised instances, and state-reading verifiers that survive UI changes. |
| Is a web agent | BrowserGym | One agent, eight benchmarks, one observation-space config — report per benchmark. |
| Needs a number for a slide | None of them | No two of these measure the same thing; build a suite over your own tasks. |
And whichever you choose, add the gold-state self-test these suites do not ship: construct each task's correct end state by hand, run the verifier against it with no agent in the loop, and drop any task whose check fails there. Every verifier that fails on a hand-made success is a task you have been scoring as a model defect.
FAQ
Can I compare an OSWorld score with a WindowsAgentArena score?
More than you can compare either with the other two, because they share a verifier library and a task-file format. But not directly: the task sets differ (368 vs 154), the applications differ, and WindowsAgentArena has a difficulty switch that changes whether the agent has to set the task up itself. Treat them as the same measuring instrument applied to two different exams.
Is BrowserGym a benchmark?
No. It is a harness with a uniform task interface over eight benchmarks, and it owns no verifier — its WebArena task imports the upstream WebArena evaluator. A result is always a result on one of the wrapped benchmarks, and an average across them mixes pass rates with partial-credit scores.
Which of these has the most reliable scoring?
AndroidWorld's verifiers are the most passive — they read device state rather than operating the device to make state readable — so they have the fewest ways to fail on a machine where the agent succeeded. That is a different claim from being the most reproducible, where its randomised instantiation works against it.
Does AndroidWorld's "millions of variations" mean it cannot be gamed?
It raises the cost of memorising specific instances, which is real. It does not prevent fitting to the 116 task families, the 20 apps, or the template phrasings, and the step budget is still a fixed target to optimise against. Anti-memorisation is a property of the instance, not of the task distribution.
Why does OSWorld download files from the network during a task?
Because the initial state has to come from somewhere, and shipping fixtures inside the VM image would make the image enormous and unversioned. It is a defensible trade. It also means 293 download steps sit in the critical path of a scored run, and a CDN failure is reported as an agent failure unless you check.
Further reading
On this wiki:
- When the Check Has to Act — why an active verifier's failures land in your agent's column, and the gold-state self-test.
- The Benchmark Landscape — where these sit among everything else being measured.
- Reading Agent Benchmarks — the general discipline for scores you did not produce.
- Computer-Use & GUI Agents — building the thing these suites measure.
- Browser Agent Failure Modes — what goes wrong inside the web lane specifically.
Project sources:
- xlang-ai/OSWorld — Apache-2.0.
- microsoft/WindowsAgentArena — MIT.
- google-research/android_world — Apache-2.0.
- ServiceNow/BrowserGym — Apache-2.0.