Evaluating Multi-Turn Agents

Step 1A fixed transcript expires at the first divergence.

Single-turn evaluation works because the input is independent of the output. You hold the prompt fixed, vary the system, and compare answers. Multi-turn breaks that: turn n's user message was a response to turn n−1's agent message, so the moment your agent's reply differs from the one in the recording, every subsequent user turn is a non-sequitur and every expected answer is grading a conversation that did not happen.

# a recorded case, and what happens when the agent improves

RECORDED                          YOUR AGENT
u: my router is offline           u: my router is offline
a: have you tried rebooting?      a: I see line sync is down
                                     upstream — this is an
                                     outage in your area
u: yes, still nothing             u: yes, still nothing     <- nonsense
a: I'll dispatch a technician
   [expected: dispatch=true]                                <- ungradeable

# the agent did BETTER and scores worse. the test set
# encodes one path through the conversation as the only path.

This is not an edge case; it is the common case, and it biases in a specific direction. A fixed transcript rewards the agent that reproduces the recorded trajectory, which is usually the agent most similar to whichever system generated the recording. Improvements that shorten the conversation — the agent that diagnoses in one turn what took the recorded agent four — look like failures, because the script still has three turns left to deliver.

You have exactly three honest options, and they measure three different things.

Step 2Replay the prefix, simulate the user, or score the end state.

• Replay and score only up to divergence. Cheap, deterministic, reproducible, and it measures a narrow thing: whether the agent stays on a known-good path. Useful as a regression detector — if yesterday's build handled turn two this way and today's does not, you want to know. Useless as a capability measure, because the score is a function of how long the agent agrees with the recording, and divergence is not failure. Report it as "turns matched before divergence," never as a pass rate.
• Simulate the user with a model. The only design that can evaluate a conversation the agent steers. You give a simulator a persona and a goal, let it talk to the agent, and grade the outcome. This is what τ-bench and its successors do, and it is the right default for anything conversational. The cost is that your measurement now contains a second model you did not intend to evaluate — STEP 3.
• Score the end state and ignore the conversation. Check the database, not the dialogue: was the booking changed, was the refund issued at the right amount, was the ticket escalated to the right queue. This is the most robust signal in multi-turn evaluation precisely because it does not care how the conversation went, and it is the one that transfers to production. Its limit is that it cannot distinguish a good conversation from an unpleasant one that happened to end correctly, and it says nothing at all about the cases whose correct end state is "no write."

Use all three, for different purposes, and keep them in separate columns. The usual mistake is to average the replay score and the end-state score into one headline number, which produces a figure that moves for two unrelated reasons. The broader argument for scoring the path separately from the outcome is in trajectory and process evaluation; what multi-turn adds is that the path is now co-authored, so path-matching metrics degrade much faster than they do on a single-turn tool-calling task.

Step 3The simulator is inside the system under test, and nobody reports it.

Once a model plays the user, the transcript is a joint sample from two policies. A weak simulator that volunteers the account number unprompted makes every agent look competent, because the hard part of the task — eliciting the information — never happens. A simulator that is too adversarial, or that answers questions the real user could not answer, makes every agent look incompetent. Neither shows up in the headline figure, and swapping the simulator model silently recalibrates every historical score you hold, exactly the way a judge upgrade does in judge calibration.

τ²-bench makes the structure explicit by modelling the setting as a decentralised partially observable Markov decision process in which both parties can talk, call tools and receive observations — a dual-control environment, rather than a user who only emits text. That framing matters because it names the thing being measured: not the agent's policy, but the pair's joint behaviour under partial observability on both sides. Its telecom domain, where the user holds tools the agent cannot reach and must be talked through using them, is reported as substantially the hardest; the paper's figures for one frontier model of the time run about 74% on retail and 56% on airline against roughly 34% on telecom. The gap is not extra task difficulty so much as the cost of having to act through somebody else.

Then there is the harness, which is where the quietest variance lives. The official τ²-bench evaluation applies a no-thought-carry-over policy: prior reasoning content is stripped between turns. A 2026 paper reporting on the same benchmark notes that this setup consistently cost 3 to 5 points relative to carrying the latest turn's reasoning forward. That is a harness switch, not a model property, and it is larger than most of the differences teams announce between models.

Step 4Per-turn reliability compounds, so capability and dependability separate further every turn.

Multi-turn is where the gap between "can do it" and "does it every time" becomes the whole story, because a conversation succeeds only if no turn breaks it. If turns fail roughly independently at rate p, a conversation of t turns completes at (1−p)^t, and the arithmetic is unkind.

# probability a whole conversation survives

per-turn   3 turns   5 turns   8 turns   12 turns
0.99        0.970     0.951     0.923     0.886
0.97        0.913     0.859     0.784     0.694
0.95        0.857     0.774     0.663     0.540
0.90        0.729     0.590     0.430     0.282

# a 2-point per-turn improvement (0.95 -> 0.97) is worth
# 15 points of completion at 8 turns. the lever is turns,
# then per-turn reliability -- never the headline average.

This is why pass^k — the probability that all k independent attempts succeed — is the metric that actually describes a deployed conversational agent, and why it collapses so much faster than pass@k flatters. An agent can post a respectable single-run score and be a coin flip on the same task repeated, and the coin-flip band is what ships. The statistics of this, including why between-task variance means more tasks beats more repeats, are worked out in eval variance and statistical power.

One multi-turn-specific caution on the independence assumption: it is optimistic in the wrong place. Turn failures correlate, because an agent that has misunderstood the user at turn two carries that misunderstanding forward — the error is in the context now. So the table above is a ceiling, and the real distribution is more bimodal than it suggests: conversations that go well go well throughout, and conversations that go wrong rarely recover. That shape is the thing worth plotting, and an average hides it completely.

Step 5The realistic hard case is the user changing their mind.

Most multi-turn test sets give the simulated user one goal and hold it fixed, which is the easy version of a conversation. Real users reverse decisions, add constraints halfway, and abandon one task for another — and a fixed script structurally cannot express that, because a goal shift only has meaning relative to what the agent has already done.

AgentChangeBench, a 2025 benchmark built for exactly this, is worth reading less for its leaderboard than for its instrument panel, which separates four things a single success rate conflates: task success, tool-use efficiency, a tool-call redundancy rate, and goal-shift recovery time — how many turns the agent takes to start acting on the new goal after the user changes it. Recovery time is the measurement teams are missing. An agent can score well on final task success while taking six turns to notice the user no longer wants the original thing, and the six turns are the entire user experience.

Redundancy is the other one worth stealing. After a goal shift, the characteristic failure is not refusal but re-litigation: the agent re-calls tools it already called, re-asks for information it already has, or half-serves both goals. Counting redundant tool calls per conversation detects that directly, and it is a cheap counter rather than a judge. Both failures are goal drift seen from the evaluation side, and the recovery-time metric is the one that makes a clarifying question look like the cheap move it usually is.

Design the shift deliberately rather than hoping the simulator improvises one: give the persona a scripted change of mind at a defined trigger — after the first successful write, or at turn four — so the measurement is reproducible. A shift the simulator invents at random is a source of variance, not a test.

Step 6Build the small version first.

You do not need a published benchmark to get most of this, and you should not use one as your gate: public multi-turn sets are not your domain, and your policies are the thing your agent gets wrong. Twenty conversations of your own, built properly, outrank a leaderboard.

• Write personas and goals, not transcripts. A persona is a few facts the user knows, a few they do not, a goal, and a tone. The information the user withholds is the design surface — the agent's job is usually to elicit it, so a persona that volunteers everything tests nothing.
• Pin the simulator and version it like code. One model, one prompt, in the repo, changed deliberately. When you do change it, re-run your history and expect every number to move.
• Grade the end state in the database. Assertions on writes, amounts and queues — not on wording. This is the part that survives a simulator change.
• Add a no-write expectation. At least a quarter of your cases should have "the correct outcome is that nothing happened" — the ineligible refund, the out-of-policy change. A suite in which every case ends in a successful write rewards an agent that always writes.
• Repeat, and report the repeat. Five runs per case and the all-of-five rate, not the best of five. If you can afford only one axis, more cases beats more repeats.
• Log recovery time and redundant calls on every run, even before you have goal-shift cases. They are counters, they cost nothing, and they are the two numbers that move first when a prompt change makes the agent stubborn.

If you cannot instrument the agent at all — a vendor product behind an API — the techniques narrow but do not vanish; evaluating an agent you can only call covers what repeat-n and side-effect assertions still buy you. And whatever you measure, a score has no denominator until you know what a competent human scores on the same conversations, which is the subject of human baselines.