BFCL v4 vs τ²-bench vs ACEBench vs ToolSandbox

The four benchmarks everyone quotes for tool calling disagree about one thing, and it is not difficulty: who else is allowed to change the world during an episode. BFCL v4 holds it still and, by its own users' account, measures argument serialisation; ToolSandbox lets the tools carry state; τ²-bench lets the customer operate their own device, which drops gpt-4.1 from 74% on retail to 34% on telecom; ACEBench lets the instruction be wrong. Each one can only see the failures its answer admits — so pick by the production failure you cannot afford, and quote the simulator error rate next to the score.

By Agentic AI Wiki 12 min read

A tool-calling score is meaningless without knowing who else was allowed to act. These four benchmarks differ on exactly that, and the cost of the difference is measurable: in the τ²-bench paper, gpt-4.1 scores 74% on retail, 56% on airline and 34% on telecom — and telecom is simply the domain where the customer also holds tools. Nothing about the model changed. If you are choosing a number to quote, or an eval to build against, choose by which degrees of freedom you are willing to give the environment.

At a glance

Four benchmarks, four different answers to "what is the thing being graded". The fourth column is the one worth reading first, because it is what each benchmark structurally cannot see.

BenchmarkGraded objectEnvironmentBlind to
BFCL v4 (Berkeley) The call itself — name and arguments, matched against a reference Static stubs; the agentic domain adds web search (200) and memory (465) Whether the call would have worked against a real, stateful service
ACEBench The turn — credited only when every call in it is correct Normal / Special / Agent splits; LLM-based or live API execution State that persists across turns in the Normal and Special splits
ToolSandbox (Apple) The trajectory — milestones hit, minefields avoided Stateful tools with implicit dependencies; a simulated user Partial credit beyond the milestone list the authors chose
τ²-bench (Sierra) The end state of a shared world, over k independent runs Dual control — the user operates device tools, the agent account tools Which of the two parties caused a failure
Who else may change the world during an episode, by benchmark Four tool-calling benchmarks scored on three properties: whether the tools themselves hold state across calls, whether the simulated user can take actions of its own, and whether the grader is stochastic. BFCL v4's agentic domain has none of the three. ACEBench holds state only partly and introduces a model-driven grader. ToolSandbox has stateful tools with implicit dependencies and a simulated user that can end the conversation. The telecom domain of tau-squared-bench adds a user that operates its own device tools. Degrees of freedom in the environment TOOLS HOLD STATE USER CAN ACT GRADER VARIES BFCL v4 (agentic) No No No ACEBench Agent split only No LLM or live API ToolSandbox Yes, implicit Ends the call Simulator τ²-bench (telecom) Yes Holds own tools Simulator held still partly free free to move
Read down the columns, not across the rows: each yes is a class of failure the benchmark can now see, and a source of variance in the score.

BFCL v4 — the honest one about what it measures

The Berkeley Function Calling Leaderboard is the number you will see quoted most, and it is the one whose scope is most often overstated. Its V4 agentic domain is genuinely harder than the single-turn AST categories it grew out of: web search with multi-hop reasoning and error recovery (200 cases), agent memory management (465 cases, with results stored per memory backend), and a format-sensitivity track built on the premise that a model should produce the same correct output when the same query arrives in a different shape.

The format-sensitivity result is the useful one

That last track is an unusual thing for a leaderboard to measure and the most directly actionable finding in the suite. BFCL varies the function documentation style — Python, XML, JSON — and reports that smaller models did better when asked to return calls in Python or JSON than in XML. If you are serving a small model and your harness happens to use XML tool blocks, that is a number about your stack rather than about the field.

What the echo stubs cost you

The caveat comes from a group that used the benchmark rather than from a sceptic. In The Bitter Lesson of Tool Calling (arXiv:2608.06370, August 2026), the authors ran a 309-entry subset of BFCL v4 across 14 models and noted that the benchmark's echo-return stubs mean it measures argument serialisation rather than end-to-end tool-use correctness. The tool returns what you sent it; nothing downstream can be wrong because nothing downstream exists.

That is not a flaw — it is the reason BFCL is cheap, fast, monthly-updated and reproducible, and it makes it the right instrument for a specific job: comparing how reliably a model emits a well-formed call. It is the wrong instrument for the question "will this agent complete a customer's task", and the leaderboard's own weighting across categories does not fix that, because every category shares the same stub-shaped world.

ACEBench — grading the instruction, not just the call

ACEBench (arXiv:2501.12851) organises its data by evaluation methodology rather than by domain, and the middle category is its contribution. Normal covers basic function calling. Agent covers multi-turn use driven through multi-agent interaction. Special covers ambiguous or incomplete instructions — which is to say, it grades whether the model declines to guess.

That matters because the most expensive tool-calling failure in production is rarely a malformed argument. It is a confidently well-formed call made on an instruction that did not contain enough information to justify it. A benchmark that only ever presents complete instructions cannot distinguish a model that asks from a model that invents, and most of them only present complete instructions.

Two mechanics to know before you quote a number from it. First, the turn-level metric credits a turn only when every call in it is correct, so a turn with three calls and one bad argument scores zero — harsher than BFCL's per-call matching and closer to how a user experiences it. Second, the overall score is a weighted combination of the three category accuracies with weights set by the square roots of their sample sizes, so a headline ACEBench number is not an average of the three things you just read about. The benchmark has also drawn criticism for relying on LLM-based or real API execution to evaluate, which is both the source of its realism and the source of its cost.

ToolSandbox — the tools remember, and only the user can hang up

ToolSandbox (arXiv:2408.04682, Apple, NAACL 2025 Findings; code at github.com/apple/ToolSandbox) was built against a specific complaint: earlier benchmarks evaluated stateless RESTful APIs from a single-turn prompt, or replayed off-policy dialogue trajectories. Its four additions are stateful tool execution, implicit state dependencies between tools, a built-in LLM user simulator that keeps the conversation on-policy, and milestone-and-minefield scoring.

Milestones and minefields

Each milestone is scored from 0 to 1 by similarity to steps in the trajectory; a trajectory that matches a minefield — an event that must not happen — scores zero outright. This is the only one of the four with a built-in notion of an unacceptable action as distinct from an incorrect one, which maps closely onto how you would actually gate an agent that can send money or delete a record.

The three hard categories are a production checklist

State Dependency, Canonicalization and Insufficient Information are where even top models struggle, and they are worth reading as a list of things to test in your own system. State dependency is the "you must enable wifi before searching" class of precondition. Canonicalization is the gap between "tomorrow at 3" and the timestamp your API wants. Insufficient Information withholds a required tool and expects the agent to recognise that the task cannot be completed — the behaviour our unsatisfiable tasks entry argues is the single most under-tested property of a deployed agent.

One design detail with outsized consequences: the simulated user is the only role that can end the conversation, through a single end_conversation tool. The agent cannot terminate the episode. That removes a whole class of trivially-scoring behaviour and it means the simulator's judgement is load-bearing — the authors are candid that it still shows hallucination and instruction-following errors, and suggest a tool-assisted simulator as future work.

τ²-bench — the user has hands

gpt-4.1 pass^1 by domain in the tau-squared-bench paper Horizontal bar chart of the pass-at-one rates reported for gpt-4.1 in the tau-squared-bench paper: retail 74 percent, airline 56 percent, and telecom 34 percent. Telecom is the domain in which the simulated user also operates tools, and it is less than half the retail rate. gpt-4.1 pass^1 by domain (%) Retail 74 Airline 56 Telecom 34 0 25 50 75 100
Same model, same metric, three domains. Telecom is the one where the customer also operates tools.

τ²-bench (arXiv:2506.07982, from Sierra) extends τ-bench's retail and airline domains with telecom, and models it as a Dec-POMDP in which both parties act on a shared, dynamic environment. The customer runs device tools — toggling airplane mode, restarting the handset — while the agent runs account tools. The agent therefore has to instruct, confirm and recover from a party it cannot control and cannot observe directly.

The headline numbers are the strongest argument in this whole comparison. The paper reports gpt-4.1 at 74% on retail, 56% on airline and 34% on telecom, measured with pass^k — the fraction of k independent runs that succeed, which is a reliability metric rather than a best-of metric. The paper also reports that moving from a No-User configuration to the collaborative default causes a substantial pass^1 drop for both models tested. Giving the environment one more actor cost more than half the retail score.

It is also the only one of the four that publishes its own grader's error rate, and that candour is worth copying. The telecom user simulator had a 16% error rate with 6% critical errors — better than τ-bench's retail simulator at 40% and 12%, and still large enough that a four-point difference between two models is noise. Note too that Amazon's τ²-Bench-Verified corrects task definitions, expected actions and evaluation criteria that did not align with the stated policies or database contents; its leaderboard scores are much higher than the original paper's and are not comparable to them. If someone quotes you a τ²-bench number, ask which one.

Cross-cutting: three ways these diverge

Where the variance lives

BFCL's variance is in the model and the format. ACEBench's is in the model plus a grader that is itself a model or a live API. ToolSandbox and τ²-bench put a stochastic agent — the user simulator — inside the environment, so two runs differ for reasons that have nothing to do with the system under test. The practical rule: on the first two, a single run is a measurement; on the second two, a single run is an anecdote, which is exactly why τ²-bench's metric is defined over k runs. See eval variance and statistical power for how many runs that implies.

What "correct" is allowed to mean

Only τ²-bench grades an end state, which is the only definition that survives an agent taking a different-but-valid route. BFCL grades the route's first step against a reference; ACEBench grades a whole turn all-or-nothing; ToolSandbox grades similarity to a chosen trajectory with hard exclusions. A model that solves a task an unexpected way is penalised by three of the four — which is a reason to be suspicious when a model that scores well on your internal end-state eval scores poorly on a public trajectory one.

How fast each one rots

BFCL's web-search track depends on a live web and so decays as pages change; its memory track and the other three are self-contained and decay only through contamination. Conversely BFCL is updated monthly and its harness is maintained in the open, while the research benchmarks are snapshots that accumulate forks — the τ²-bench situation, with an original paper, a Sierra repository and a corrected Amazon variant, is the normal end state. Benchmark contamination is the other clock running on all four.

When to pick which

Your questionUseBecauseDo not use
Which model emits well-formed calls against my schemas? BFCL v4 Cheap, monthly, and argument serialisation is exactly what it measures τ²-bench — you will pay for a user simulator you do not need
Does my agent ask instead of guessing? ACEBench Special, ToolSandbox Insufficient Information Both deliberately present tasks that cannot be completed as stated Any benchmark whose instructions are always complete
Will it survive a multi-turn task with preconditions? ToolSandbox Implicit state dependencies and canonicalization are its hard categories BFCL — the stubs have no state to depend on
Can it drive a human through a procedure? τ²-bench telecom Dual control is the only environment where the user can fail you Single-actor benchmarks, which score this at the level of the easy domains
Is this release better than the last one? Your own end-state eval, plus one of these as a regression tripwire Public benchmarks cannot see your tools, your policies or your denominator A single headline number from any of the four

FAQ

Which tool-calling benchmark should I quote?

Name the environment with the score. "87% on BFCL v4" and "87% end-state success with a user who can act" are different claims, and only the first is cheap to produce. If you have to pick one for a public claim, pick the one whose environment most resembles your product, then publish the configuration alongside it.

Why did the same model drop from 74% to 34%?

Because the telecom domain of τ²-bench gives the simulated customer their own tools. The agent must instruct a party it cannot control, observe only what that party reports, and recover when they do the wrong thing. That is a different task, not a harder version of the same one.

Does a high BFCL score mean the agent works?

It means it serialises arguments well. The Bitter Lesson of Tool Calling authors note the benchmark's echo-return stubs measure exactly that, and nothing downstream of the call. Treat it as a necessary condition.

Can I trust a four-point difference between two models?

Not on τ²-bench or ToolSandbox. τ²-bench reports a 16% user-simulator error rate with 6% critical errors in telecom; a difference smaller than the grader's own error rate is not a result. Run k trials and report pass^k.

Which τ²-bench numbers are the real ones?

Both are, for different questions. The original paper's scores are the published baseline. Amazon's τ²-Bench-Verified corrects tasks whose definitions did not match the stated policies or the database, and its scores are substantially higher — so they measure a cleaner benchmark and cannot be compared with the original's.

Should I run any of these on my own agent?

Run ToolSandbox or τ²-bench once, to see which hard category breaks you, then build the equivalent against your own tools. The public task sets will never contain your schemas or your policies, and the point of the exercise is the failure taxonomy, not the leaderboard position.

Further reading

On this wiki:

Project sources: