Evaluation & Observability

Why Evaluating Agents Is Hard

Non-determinism, compounding multi-step error, no single gold answer, path-dependence, eval cost, and dataset rot — the six reasons one clean number is a lie.

Online vs offline evals

Offline evals catch regressions before deploy; online evals catch the user behavior you couldn't fake — why you need both, and where each one lies to you.

Outcome vs Trajectory Evaluation

End-state predicates vs grading the decision sequence: when each is right, partial credit, and tool-call assertions as the highest-leverage safety check.

LLM-as-Judge for Agents

Rubric design, pairwise vs pointwise, the biases that invert verdicts, calibrating against human labels, and the cases where you must not use a judge.

Reading Agent Benchmarks Critically

What SWE-bench, GAIA, τ-bench and WebArena actually measure, why contamination and harness sensitivity make rank a weak signal, and the small custom set that really decides.

Tracing & Observability for Agents

The trace is the data structure, not a log: what to record per step, spans and OpenTelemetry GenAI conventions, and trajectory replay as the bridge to eval.

Eval-Driven Agent Development

The eval is the only spec an agent has: tiered CI gates, golden trajectories, offline vs online, the production-to-eval flywheel, and the no-regression ratchet.

OpenTelemetry GenAI Semantic Conventions

Instrumentation is a data-model decision, not a dashboard one: the agent/workflow/tool/model span kinds, why Development status argues for pinning rather than waiting, splitting structural telemetry from prompt content at the collector, and the one collector hop that makes every later vendor choice reversible.

Detecting Quality Regressions

Production has no labels and a judged metric needs ~1,400 scored runs to see a five-point drop, so the detector is the shape of the run — step-cap rate, per-tool errors, termination mix — and the judge is only the confirmation.

Production Feedback Signals

Thumbs arrive from under one percent of sessions and reward confidence over correctness, while the diff between the agent's output and what the user actually shipped is a dense, free, expert-written label — treat every feedback signal as a router into the eval set rather than a metric to optimise.

Annotation & Labeling Ops

A judge cannot be more accurate than the labels it was calibrated against, so if two of your experts agree on 72% of traces a judge at 72% is already at the ceiling — measure inter-annotator agreement first, read low agreement as a rubric defect, and route disagreement to adjudication instead of averaging it away.

Trace Sampling & Retention

Sampling 10% at run start keeps 10% of your failures, and nothing at step zero predicts which run goes wrong — so buffer to run end, keep every failed, capped and expensive run whole, cut successes hard, and treat retention as a decision about the eval set you have not built yet.

Simulated Users in Agent Evaluation

Every multi-turn agent score measures two systems, and the second one is an unversioned model playing a customer that can move your number several points on its own — pin its model ID, prompt and seed like a dependency, calibrate it against real transcripts, never let it judge whether it was satisfied, and give graders an explicit simulator-fault verdict.

Measuring Agent Latency

A fifteen-step trajectory turns a one-in-a-hundred slow call into a one-in-seven slow task, so the p99 of a step predicts your users' experience far better than its p50 — measure the trajectory rather than the call, split model from tool from queue time, and keep time-to-first-useful-output separate from time-to-done.

Maintaining an Eval Set

An eval set decays by being fitted, not by rotting: every regression you fix converts a discriminating case into a permanent pass, so measure the fraction of cases all candidates already pass, score cases by how often they changed anyone's mind, and run a standing replacement rate instead of a periodic cleanup.

Online Experiments for Agents

Detecting a three-point lift in task success takes about 3,700 sessions per arm before anything agent-specific, and clustering by user typically triples it — so randomise the user rather than the request, pre-commit to one decision metric chosen for its variance, and when the arithmetic says the test is unaffordable, run a guarded rollout and label it as one.