Evaluating Agents

Why an agent is not a single model call.

In evals-101 an eval was three things in a list: an input, an expected behavior, and a scoring rule. That model fits a single model call perfectly — one prompt in, one output out, one thing to grade. An agent breaks it. Running the agent loop doesn't produce one output; it produces a trajectory: a branching, stateful sequence of reasoning steps, tool calls, and observations that unfolds over many turns.

Two properties of that trajectory make agent eval genuinely different from scoring a chat completion:

• Many valid paths. The same task can be solved correctly by very different sequences of actions. There is rarely one gold trajectory to diff against — a good eval has to accept a class of acceptable paths, not one canonical script.
• Non-determinism and state. Two runs of the same agent on the same input can diverge — different tool results, different sampling, different intermediate state carried forward. A single pass tells you little; you have to run each case several times and reason about the distribution, not one lucky or unlucky sample.

So this entry is the agent-specific follow-on to evals-101: the small-trusted-set discipline still holds, but the unit under test is a multi-step, stateful, non-deterministic process — and that changes what you measure and how.

Outcome eval, trajectory eval, and the judge that scores them.

There are two complementary lenses, and mature teams use both:

• Outcome (final-answer) eval. Did it reach the right end result? Cheap to run and easy to score, but blind to how the agent got there.
• Trajectory (process) eval. Scores the steps themselves: did it call the right tool, with the right arguments, in a sensible order, without wasteful or unsafe actions? This catches failures outcome eval cannot — a step that produced no visible harm here but would in production, or a right answer reached by a broken or lucky path.

For open-ended agent output — a written answer, a plan, a summary — there is often no code-checkable rule, so the dominant technique is LLM-as-a-judge: a second LLM grades the output against a rubric. It is genuinely useful and it scales, but it is not ground truth, and it has well-documented biases you must design around:

• Position / order bias — when comparing two candidates, judges tend to favor whichever came first. Mitigate by swapping the order and averaging both directions.
• Verbosity / length bias — judges tend to rate longer answers higher regardless of quality. A tight rubric that scores substance, not word count, blunts this.
• Self-preference bias — a judge tends to over-rate output written in its own style or by its own model family.

The fix is not to abandon the judge but to calibrate it: write clear rubrics, randomize order, and check the judge's scores against a set of human labels so you know how far to trust it. That calibration loop — and how to meta-evaluate the judge itself — is the subject of judge calibration and meta-evaluation.

Your small custom set beats the leaderboard — and it must be multi-turn.

Reading benchmarks critically already made the general case; here is the agent-specific edge of it. Static public benchmarks under-predict how your agent behaves in production, because leaderboards run fixed, often-contaminated, generic tasks — while your app has its own tools, its own data, its own policies, and its own failure modes. Named benchmarks such as SWE-bench (and the curated SWE-bench Verified) for coding agents, GAIA for assistants, τ-bench for multi-turn tool use, and AgentBench are excellent reference points for the field, but a high rank on any of them is not proof your agent works for you. A small, custom, trusted eval set built from your own real tasks is the thing that actually decides that — the same lesson evals-101 teaches, now applied to trajectories.

There is one more agent-specific requirement. Real use of an agent is a conversation, not a single prompt-and-response, so your eval has to be too. The standard technique is a simulated user — often another LLM — that plays the human and pushes the agent through a multi-turn scenario, testing whether it stays coherent across turns and whether it adheres to your policies under pressure. τ-bench is the well-known public example of exactly this shape: multi-turn, simulated-user, policy-adherence scoring. You want the same shape pointed at your policies.

The scoreboard has four columns, and it runs in CI.

Accuracy alone is the wrong scoreboard for an agent. The metrics that matter are multi-dimensional, and you read them together:

• Task success rate — did it actually accomplish the task, judged over multiple runs.
• Cost — tokens and dollars per task. This is now a first-class eval axis, not an afterthought.
• Latency — how long a task takes end to end.
• Safety — did it ever take a forbidden or destructive action, the concern that guardrails-101 is built around.

Read together, these overturn the leaderboard instinct. A cheaper agent that succeeds roughly 80% of the time at a tenth of the cost can beat one that succeeds 85% — that is the cost, quality, latency trade-off applied to eval, and it is why cost belongs on the scoreboard next to accuracy.

Finally, treat all of this like tests, not vibes. In eval-driven development, the eval suite is version-controlled, runs on every prompt, model, or tool change to catch regressions before deploy, and grows every time a production failure is found — each new bug becomes a permanent case so it can never silently return. Platforms such as LangSmith, Braintrust, Arize Phoenix, and the open-source OpenAI Evals framework exist to run these suites in CI. When you are ready for the depth behind this entry — how public agent benchmarks are saturating (see AgentBench and the newer landscape), and how leaderboards like HAL now plot accuracy against cost — read the benchmark landscape (2026) and HAL and asynchronous agent eval essays in the Evaluating Agents deep-dive group.