Online Experiments for Agents

Do the sample-size arithmetic before you write the variant.

Every argument on this page follows from one calculation that takes thirty seconds and is almost never done. For a binary outcome — task succeeded or it did not — the sessions needed per arm at 80% power and a 5% two-sided significance level are:

n ≈ (z(α/2) + z(β))² × 2p(1-p) / δ²
  = (1.96 + 0.84)² × 2 × 0.70 × 0.30 / δ²

  δ = 0.03  (70% → 73%)   n ≈ 3,700 sessions per arm
  δ = 0.05  (70% → 75%)   n ≈ 1,300 sessions per arm
  δ = 0.10  (70% → 80%)   n ≈   330 sessions per arm

Three things follow immediately. Small effects are unaffordable: halving the effect size quadruples the cost of seeing it, so a change you believe is worth two points is not something an online test will confirm. A shipped agent doing 200 sessions a day needs about five weeks per arm to resolve three points — during which you will have changed the model, the prompt and a tool. And if your outcome is scored by a judge rather than observed, the judge's own disagreement with human labels adds variance on top of this, which is the same arithmetic that drives quality regression detection.

Run the numbers first and most experiments are decided before they start — not by the result, but by whether the result could ever have arrived.

Randomise the user, not the request.

Per-request randomisation is tempting because it maximises sample size, and it is wrong for agents in three specific ways. Agents carry memory, so a user's session in the treatment arm changes what the control arm sees next. Users adapt — someone who learns the agent handles multi-part requests well starts sending them, and that behaviour follows them into whichever arm they land in tomorrow. And a single task frequently spans several sessions, so per-request assignment splits one job across both arms and measures neither.

Assign at the level at which the memory and the human live: the user, or for a B2B agent the tenant. Then pay the clustering cost honestly, because sessions from one user are correlated:

design effect = 1 + (m - 1) × ICC

  m   = sessions per user in the window   (say 20)
  ICC = within-user correlation           (0.05-0.15 is typical)

  deff = 1 + 19 × 0.10 = 2.9
  3,700 sessions per arm  →  ~10,700 sessions  ≈  535 users per arm

Analysing clustered data as if it were independent is the single most common way an agent experiment reports significance it does not have — it shrinks the standard error by roughly the square root of the design effect, which is to say it invents a result. If you have fewer than a few hundred distinct users per arm, no amount of session volume rescues the test, and tenant-level assignment brings its own arithmetic: ten enterprise customers cannot be randomised into two arms at all, which is why multi-tenant agents usually need a within-tenant design or none.

Choose one decision metric, by variance, and keep the rest as guardrails.

Task success is the metric everyone wants and the most expensive one to measure, because it is binary, often delayed, and frequently unobservable without a judge or a human. Cheaper metrics are not worse metrics — they are metrics with smaller variance per session, which converts directly into a shorter experiment:

• Steps or tool calls to completion. Continuous, observed on every run, and usually the thing the change was actually meant to improve.
• Escalation or handoff rate. Binary but often further from 50%, and it is a business metric rather than a proxy for one.
• Edit distance between what the agent produced and what the user shipped. Dense, free, expert-written, and the argument for it is in production feedback signals.
• Retry and re-ask rate within a session. The most sensitive early indicator that a change made the agent worse, and it moves within hours rather than weeks.

Pick exactly one as the decision metric and pre-commit to it. Everything else — cost per task, p95 trajectory latency, safety events, refusal rate — is a guardrail: it can stop the rollout, but it cannot be promoted to the win condition after the fact. Two decision metrics is not a rigorous experiment; it is two chances to declare victory. Latency in particular must be read at the trajectory level rather than the call level, for the reason set out in measuring agent latency.

Peeking is not free, so buy the sequential test.

Everyone watches the dashboard daily and stops when it looks good. Under a fixed-horizon test that behaviour is not impatience, it is a change to the false-positive rate: five looks at a nominal 5% threshold produces a real error rate closer to 15%, and a team that runs twenty experiments a year this way ships several changes that did nothing.

There are only three honest options. Fix the horizon in advance and do not look at the decision metric until you reach it — feasible, unpopular, and best paired with an automated report so nobody has to be trusted. Use a sequential method designed for continuous monitoring — group-sequential boundaries or always-valid confidence intervals — which costs roughly 10–20% more samples for the right to stop early and look whenever you like. Or use the dashboard for guardrails only, monitoring safety and cost continuously while the decision metric stays sealed.

Variance reduction is cheaper than traffic.

If you cannot get more users, get more information out of the ones you have. Three techniques transfer cleanly from web experimentation to agents and are underused here:

• Pre-period covariates (CUPED). Adjust each user's outcome by their own behaviour in the weeks before the experiment. Variance falls by a factor of 1 - ρ² where ρ is the correlation between the pre-period and in-experiment metric; a modest ρ of 0.5 removes a quarter of the required sample. Agent users are strikingly self-similar week to week, so ρ is often better than that.
• Stratification. Randomise within task type, tenant size or locale rather than across everything. A mixed traffic stream where 20% of tasks are hard and 80% trivial has most of its variance in the mix, not in the treatment.
• Paired offline replay alongside the live test. Run both variants over the same recorded trajectories with the same fixtures and seeds. It answers a different question than the live test — it cannot see user adaptation — but it answers it with paired data at a fraction of the cost, which is exactly the division of labour described in online vs offline evals.

Resist the fourth technique everyone reaches for first: narrowing the population until the effect looks big. Testing only on your most engaged users does produce significance sooner, and it produces it for a population that is not the one you are shipping to.

When the experiment is unaffordable, run a guarded rollout and say so.

Most agent teams do not have the traffic for a properly powered test on most changes, and the correct response is to stop pretending. A guarded rollout is a different instrument with a different claim: it does not establish that the change helped, it establishes that it did not obviously hurt.

• Gate on the offline eval set first — this is the only step that measures quality, and it is why the eval set is the real spec.
• Shadow the new variant against live traffic without serving it, comparing trajectories rather than outcomes.
• Ramp 1% → 5% → 25% → 100%, with each stage long enough to cover a full weekly cycle of task types.
• Attach automatic rollback to guardrails only: error rate, escalation rate, cost per task, p95 latency, safety events. The mechanics belong to rollout and versioning and the switch to feature flags.
• Record in the changelog that this was a rollout, not a test. Six months later somebody will cite "the 4% improvement we measured", and it needs to be findable that no such measurement happened.

Related: eval variance and statistical power for the same arithmetic applied offline, simulated users for the second system every multi-turn number measures, and the cost of evaluation for what all of this consumes.