Simulated Users in Agent Evaluation

You cannot evaluate a conversational agent without one.

Static fixtures work for a single turn: prompt in, answer out, grade it. They stop working the moment the agent's second action depends on what the user says in response to the first. A recorded transcript cannot answer a question the agent asks that the original user was never asked.

• The choice is not whether to simulate, it is whether the simulator is explicit. Teams that avoid one end up with a hidden equivalent: a fixed script that the agent silently fails against because it went off-path, scored as a task failure when it was a fixture failure.
• This is the design behind τ-bench and its successors — an agent talking to a simulated customer while calling domain APIs, graded against a written policy it must keep obeying. That structure is now the standard shape for any benchmark that claims to measure agents rather than completions.
• Which means the published numbers inherit the problem too. A leaderboard's agent ranking is conditional on its simulator, and two benchmarks with different simulators are not comparable even where the task list overlaps — one of the reasons in reading agent benchmarks critically.

The score is joint. Pin the simulator like a dependency.

If a simulated user is more forthcoming this quarter than last, your agent gets fewer chances to ask a bad clarifying question and its score rises. Nothing about your agent changed. This is not a subtle effect and it is the single most common cause of an agent evaluation that drifts without an explanation.

• Version everything that shapes the simulator's behaviour — model ID, temperature, seed, persona prompt, the policy document, and the stopping rule. Store them alongside the results, not in someone's notebook.
• Change one at a time. Upgrading the simulator's model in the same release as an agent change makes the result uninterpretable. If you must upgrade, re-baseline the unchanged agent first and publish both numbers.
• Never use the same model for the agent, the simulator and the judge. Shared blind spots correlate all three: the simulator asks in the phrasing the agent handles best, the judge accepts the reasoning it would itself have produced. Split at least the judge, per LLM-as-judge for agents.
• Budget for the variance. Two stochastic systems in series widen the distribution, so a difference that would be significant against fixtures may be noise here. Run more seeds and read the interval, not the point — eval variance & statistical power.

Know the four ways a simulator lies to you.

Simulated users fail in characteristic directions, and each one biases your score in a predictable way. All four are visible in transcripts and invisible in aggregates.

• Too cooperative. It answers clearly, never contradicts itself, never goes quiet, and accepts the first solution offered. Your agent looks production-ready and then meets a real customer who says "no, the other one" without saying which.
• Answer leakage. The persona prompt contains the goal state, and the simulator hands it over when pressed. The agent scores well on a task it never had to solve, and the leak is only findable by reading the conversation.
• Persona drift. Around turn eight the simulated user starts behaving like a helpful assistant — offering suggestions, summarising, being reasonable — because that is what its base model was trained to be. Long-horizon tasks are exactly where this matters most.
• Adversarial overshoot. Turned up for robustness testing, the simulator becomes a user nobody has ever met. The failures it produces are real failures against an unreal distribution, and fixing them costs capability elsewhere.

The generalisation: a simulator's job is to be representative, not to be good. Optimise it toward your real traffic, and accept that its own conversational quality is irrelevant.

Calibrate against real transcripts before you trust a number.

A simulator earns credibility the same way a judge does — by being checked against ground truth on a sample, and re-checked when anything changes. The ground truth here is your own production conversations.

• Sample real transcripts and characterise them. Turns per resolution, how often the user changes their mind, how often they supply an entity incorrectly, how often they abandon. These are your target distributions.
• Score the simulator against them. If real users take eleven turns and the simulator takes four, it is not modelling your users; it is modelling a cooperative reader of your persona prompt.
• Build personas from clusters, not from imagination. The confused first-timer, the expert in a hurry, the user with a wrong account number — derived from real traffic, with their relative frequencies preserved, so the aggregate score means something.
• Re-calibrate on a schedule. Your users change, your product changes, and the simulator does not notice either. This is the same operational discipline as annotation & labeling ops, applied to a synthetic annotator.

Score the agent, not the pair.

A single pass/fail per conversation hides which of the two systems failed, which is the one thing you needed to know. Separate the axes and the simulator's contribution becomes visible.

• Outcome — did the end state match the goal — checked by a program against the environment, never by asking the simulator whether it was satisfied. A satisfied simulated user is the least trustworthy signal in the run.
• Policy adherence — did the agent break a written rule on the way. This is the failure a happy outcome hides, and the reason trajectory evaluation earns its cost here.
• Consistency across repeats. Report success on the same task k times running rather than best-of-k. An agent that succeeds three times in five is not a 60% agent in production; it is an agent that will fail two customers.
• A simulator-fault category. Give graders an explicit "the simulated user made this unwinnable" verdict and track its rate. If it exceeds a few per cent, fix the simulator before reading any other number.

Know when not to simulate at all.

Simulation is the right tool for pre-deployment regression on multi-turn behaviour. It is the wrong tool for several things teams routinely use it for.

• Not for absolute quality claims. "94% task success" against a simulator is a statement about the pair. Report it internally, never externally, and never to a regulator.
• Not as a substitute for production signal. Real abandonment, real escalation and real repeat-contact rates measure things no simulator generates — the argument in production feedback signals.
• Not for safety-critical acceptance. If a failure mode has a regulatory or physical consequence, it needs a human-run scenario, not a synthetic one.
• Not where a fixture would do. Single-turn extraction, classification and formatting are cheaper and more stable to test with static cases. Reach for a simulator only when the branching is the thing under test.

Related: why evaluating agents is hard for the six underlying reasons, judge calibration & meta-evaluation for the same discipline applied to the grader, and agent evaluation for the concept-level introduction.