Evaluating Agents

Judge Calibration & Meta-Evaluation

Prometheus 2, JudgeBench, RubricEval; meta-evaluation collapse; the 85-90% human-agreement floor; monthly recalibration cadence.

Benchmark Landscape (2026)

SWE-bench Verified saturation (five models within 0.7 pts); SWE-bench Pro; contamination as legal deterrent; why Verified is now an audit signal, not a ranking.

HAL & Asynchronous Agent Eval

Princeton HAL (cost-per-solve + 5-dim reliability dashboard); Gaia2 (async environments, write-action verifiers, temporal constraints); why static benchmarks miss real deployment.

Trajectory & Process Evaluation

Scoring how the agent worked, not just the final answer: outcome vs trajectory eval; AgentEvals match modes (strict/unordered/subset/superset); reference-based vs reference-free LLM-judge; the process-reward-model crossover; and why exact-match on the path fails correct agents.

Eval-Driven Development & Regression Evals in CI

Evals as CI gates, not one-offs: the golden set as a living asset; pass^k and paired significance tests for non-determinism; online vs offline, canary + drift detection; cost/latency as gate-able budgets; promptfoo, DeepEval, Inspect.

Eval Variance & Statistical Power

A single-run agent score is a sample, not a measurement: pass@k vs pass^k, why between-task variance means more tasks beats more runs, paired McNemar designs that halve the detectable effect on the same budget, and the decision rule that stops teams ratcheting on noise.

Benchmark Contamination & Leakage

Contamination belongs to the (model, benchmark, date) triple, not to the benchmark: verbatim vs solution vs indirect leakage, why agent benchmarks leak the environment and not just the answer, four detection tests you can run from outside, and the rule that public scores screen while only post-cutoff data decides.