Deep-Dives / Evaluating Agents

Evaluating Agents

The 2026 discipline of evaluating agents: benchmark saturation, judge calibration, drift detection, and the eval methodologies that survived contact with production.

  1. Judge Calibration & Meta-Evaluation
    Prometheus 2, JudgeBench, RubricEval; meta-evaluation collapse; the 85-90% human-agreement floor; monthly recalibration cadence.
  2. Benchmark Landscape (2026)
    SWE-bench Verified saturation (five models within 0.7 pts); SWE-bench Pro; contamination as legal deterrent; why Verified is now an audit signal, not a ranking.
  3. HAL & Asynchronous Agent Eval
    Princeton HAL (cost-per-solve + 5-dim reliability dashboard); Gaia2 (async environments, write-action verifiers, temporal constraints); why static benchmarks miss real deployment.
  4. Trajectory & Process Evaluation
    Scoring how the agent worked, not just the final answer: outcome vs trajectory eval; AgentEvals match modes (strict/unordered/subset/superset); reference-based vs reference-free LLM-judge; the process-reward-model crossover; and why exact-match on the path fails correct agents.
  5. Eval-Driven Development & Regression Evals in CI
    Evals as CI gates, not one-offs: the golden set as a living asset; pass^k and paired significance tests for non-determinism; online vs offline, canary + drift detection; cost/latency as gate-able budgets; promptfoo, DeepEval, Inspect.
  6. Eval Variance & Statistical Power
    A single-run agent score is a sample, not a measurement: pass@k vs pass^k, why between-task variance means more tasks beats more runs, paired McNemar designs that halve the detectable effect on the same budget, and the decision rule that stops teams ratcheting on noise.
  7. Benchmark Contamination & Leakage
    Contamination belongs to the (model, benchmark, date) triple, not to the benchmark: verbatim vs solution vs indirect leakage, why agent benchmarks leak the environment and not just the answer, four detection tests you can run from outside, and the rule that public scores screen while only post-cutoff data decides.
  8. Eval Integrity & Scorer Gaming
    A score is produced by software the agent under test can reach: task-level reward hacking, harness compromise and cross-run contamination are three different failures wearing one word, and the third silently fakes the sample rather than one run. Grade out of process from an immutable transcript, delete every shared writable surface between runs, and publish the isolation properties next to the number.
  9. Evaluating Against Live Systems
    An eval that transacts with systems you do not own is a deployment with no change control, and the September 2026 Medicare-portal incident happened inside one: score a circumvented refusal as a failure or your successful trajectories become a reward for circumvention; then cap per-host load, run from an attributable egress, and keep request logs joined to trajectories with a named owner who may notify.
  10. Human Baselines in Agent Evals
    A score has no denominator until you know what a competent person scores on the same tasks, under the same cap, with the same tools, graded by the same grader — and that number was never collected for almost any benchmark teams quote. A baseline is that four-part tuple, the grader biases the comparison in both directions at once, and METR's time-horizon method is the field's only calibrated answer. Then build your own in two person-days and report the ratio, never the score.
  11. Budget-Matched Evaluation
    Two scores produced at two different budgets cannot be ranked, because accuracy-versus-budget curves cross: one model was reported at 80.1% on Terminal-Bench 2.1 using about 25,000 tokens an attempt against 81.0% using about 80,000 — a tie on the leaderboard and nothing like a tie at any cap you would deploy under. Cost-per-solve records what a submission chose to spend; budget-matching constrains what every candidate may spend, and only the second answers a deployment question. Derive the cap from your latency and cost ceilings, score truncation as failure, read the truncated share before the pass rate, and keep one uncapped arm to tell you whether the cap or the model is the ceiling.
  12. Evaluating an Agent You Can Only Call
    Without a trajectory you cannot separate a right answer from a lucky one, and that is the whole job of an agent eval — so three cheap instruments carry it instead: repeat-n to measure how often the verdict changes (report all-of-n, not pass@n: 80% per run over a five-call journey completes one time in three), counterfactual probes to establish grounding, and assertions on the side effects in systems you own. The highest-yield single run is the deletion probe: remove the supporting document from a sandbox corpus and re-ask, because an agent that still answers confidently was never reading your data. And because every technique needs an access right that is free before signature and expensive after, the eval you can run decides the contract you can sign — a version identifier, a mutable sandbox, per-request step and token counts, a cadence to run your suite, and exit terms for your eval set.
  13. Evaluating Multi-Turn Agents
    A recorded conversation expires at the first turn the agent deviates, because turn three's expected answer assumed turn two's reply — and the bias runs against the agent that diagnoses in one turn what the recording took four to reach. Generate the user instead and the simulator joins the system under test: τ²-bench models it as a dual-control Dec-POMDP where both sides hold tools, its hardest domain drops one frontier model of the time from ~74% retail to ~34% telecom, and the official harness's no-thought-carry-over policy alone is reported to cost 3–5 points. So publish the simulator, its prompt, the carry-over policy and the turn limit beside every number; grade the end state in the database; make a quarter of your cases end in no write; and log goal-shift recovery time and redundant tool calls, the two counters no leaderboard will hand you.
  14. When the Check Has to Act
    A verifier that must change the system to observe it has its own failure rate, and that rate is reported in your agent's column. OSWorld's spreadsheet tasks press Ctrl+S through pyautogui and sleep 0.5s before comparing a gold file fetched over the network, after demanding a window title match with strict: true — and across its task files the grading path issues 358 execute, 307 sleep, 293 download and 200 activate_window steps against 368 tasks. The noise is anti-correlated with capability, because only an agent that finishes reaches the postconfig. Fix it with a per-sweep gold-state replay and a third outcome: report INDETERMINATE separately and keep it out of the denominator.
  15. Gold Runs & the Unsolvable Task
    Every score is a fraction whose denominator you assumed, and tasks die quietly — a transitive dependency stops resolving, a base image tag moves, a 600-second cap no longer fits. On a worked 180-task suite, twelve dead tasks turn an unchanged agent's 62.2% into 57.8%, a 4.4-point regression that is 61.9% against the achievable denominator and therefore nothing at all. Dead tasks only ever subtract, so the drift has a sign and always reads as a model regression; a weakened test biases the other way and leaves the reference run green, which is why you need two sentinels — gold for the ceiling, a no-op solver for the floor. Terminal-Bench's maintainers are the only team to have published the rate: 26–28 of 89 expert-reviewed tasks repaired six months after release.