Trajectory & Process Evaluation

The failure outcome eval is blind to.

Outcome eval takes the final answer or end state, hands it to a grader, and checks whether it matches — that is the entire surface it sees. Trajectory (process) eval takes the ordered sequence instead: the reasoning turns, the tool-call selections, the arguments passed, the observations returned, and the decisions made in between, and it scores the shape of the work. In practice a trajectory is not an exotic artifact — it is the message/step list your framework already emits, the same list the agent appends to on every iteration of the agent loop. LangGraph exposes a second, coarser view of the same run — the graph's node-visit sequence — and some evaluators score that instead of the raw messages.

trajectory  —  task: "cancel my most recent order, then confirm"
--------------------------------------------------------------------
1  reason   "look up the user's orders first"
2  tool     get_orders(user_id="u_8841")          -> 3 orders
3  tool     get_orders(user_id="u_8841")          -> 3 orders   [redundant]
4  reason   "most recent is order_5567"
5  tool     cancel_order(order_id="order_5567")    -> {status: "ok"}
6  tool     cancel_order(order_id="ORDER_5567")    -> error: not_found
7  answer   "all set, your most recent order is cancelled"
--------------------------------------------------------------------
outcome eval     PASS   final answer matches the reference string
trajectory eval  FAIL
  step 3  redundant get_orders, no new information   (efficiency)
  step 6  hallucinated retry after success, bad id   (grounding)

The agent above returns the reference string and passes outcome eval; the trajectory tells a different story. Step 3 is a duplicate lookup that adds no information. Step 6 is a hallucinated retry that fired after the cancellation had already succeeded, against a mangled order id that happened not to matter only because the first call did the work. Outcome eval is structurally blind to all of it — it cannot see the loop, the wasted call, the wrong-cased argument, or the fact that the agent's "confirmation" was a retry that errored.

This matters beyond aesthetics. A right answer reached through a broken path is a right answer that will not survive a small perturbation of the input: the lucky retry becomes a wrong answer the day the first call fails, and the ignored hallucinated argument becomes load-bearing the day the tool stops being forgiving. It is the same distinction the training literature draws between process and outcome supervision (see process vs outcome rewards): scoring only the endpoint rewards any path that reaches it, including the ones that reached it by accident.

The step-level metric taxonomy.

Once you accept that the path is worth scoring, the question is what to score. Five step-level metrics have stabilized across the tooling, and they line up one-to-one with the failures in the STEP 1 trace.

• Tool-selection accuracy — for each sub-goal, did the agent reach for the right tool? Wrong-tool errors are the most common and the cheapest to detect, and they are the failure that tool calling makes visible in the call stream.
• Argument / parameter correctness — were the arguments grounded in the context and prior observations, or hallucinated? A user_id invented rather than read from the session is a grounding failure even when the call happens to succeed.
• Efficiency / convergence — how many steps did it take against how few it needed? Arize's convergence eval formalizes this: measure the agent's step count against a known minimum step count for that query type; redundant calls, detours, and loops show up as excess steps.
• Recovery / adaptivity — after a failed call or a bad observation, does the agent correct course or repeat the mistake? The step-6 retry is a recovery failure: it re-fired instead of recognizing the success it already had.
• Per-step grounding / faithfulness — is each step faithful to the observations gathered so far? A step can be factually true in the abstract yet unfaithful to what the trajectory actually retrieved — the classic "right for the wrong reason."

The academic framing that has caught on compresses these five into three dimensions — efficiency, hallucination, and adaptivity — which is the TRACE taxonomy (arXiv 2510.02837). Efficiency subsumes convergence; hallucination covers both wrong tools and ungrounded arguments; adaptivity is recovery. The value of the three-way cut is that it maps cleanly onto what a reference-free judge can be asked to score without a gold path.

Reference-based vs reference-free.

The cheapest, most deterministic trajectory eval compares the agent's path to a gold/reference path. AgentEvals (langchain-ai/agentevals) is the reference implementation. create_trajectory_match_evaluator() takes a trajectory_match_mode: strict (same messages, same order, same tool calls), unordered (same tool calls, any order), subset (agent called only tools that appear in the reference — an efficiency gate), and superset (agent called at least the reference tools). Argument comparison is a separate axis, tool_args_match_mode ∈ {exact (default), ignore, subset, superset}, with per-tool overrides via tool_args_match_overrides.

# agentevals — reference-based trajectory matching (deterministic, no LLM).
from agentevals.trajectory.match import create_trajectory_match_evaluator

# "subset": agent may call FEWER tools than the reference but no EXTRA ones,
# which turns the matcher into an efficiency gate (redundant calls -> fail).
evaluator = create_trajectory_match_evaluator(
    trajectory_match_mode="subset",       # strict | unordered | subset | superset
    tool_args_match_mode="exact",         # exact | ignore | subset | superset
    tool_args_match_overrides={
        "search": "ignore",               # semantically-equal queries still pass
        "get_orders": ["user_id"],        # compare user_id only, ignore paging args
    },
)
result = evaluator(outputs=agent_messages, reference_outputs=gold_messages)
# -> {"key": "trajectory_match", "score": True/False}

# reference-free — an LLM judge reads input + tool calls + observations,
# rubric-scores the path, and needs NO gold trajectory.
from agentevals.trajectory.llm import create_trajectory_llm_as_judge
judge = create_trajectory_llm_as_judge(model="openai:o3-mini")

# graph variant — score the NODES visited, not the messages.
from agentevals.graph_trajectory.llm import create_graph_trajectory_llm_as_judge

The mode choice is the whole game. strict is right only when there is exactly one correct path; for anything with legitimate ordering freedom it fails correct agents (more in STEP 5). subset is the efficiency-enforcing mode — it passes an agent that used a subset of the reference's tools and fails one that made extra calls. Reference-based matching is deterministic and free of LLM cost, but it needs annotated gold trajectories, which are expensive to produce and brittle when the task admits many paths.

When you cannot enumerate a gold path — open-ended tasks, exploratory agents — a reference-free judge reads the trajectory (the input, the tool schemas, the ordered tool calls, the observations) and rubric-scores it with no gold path to compare against. AgentEvals ships create_trajectory_llm_as_judge(); the graph variants create_graph_trajectory_llm_as_judge() and graph_trajectory_strict_match() score the LangGraph node-visit sequence instead of messages. Arize Phoenix does the same as an OTel-native, self-hostable eval: an LLM judge classifies the ordered tool-call sequence as correct or incorrect with an explanation, against a rubric that asks whether the path progresses logically, uses the right tools, and stays reasonably efficient with no unnecessary detours — Phoenix names these the tool-calling eval and the convergence/path eval, and calls the ideal route the "golden path." Braintrust adds trajectory-level scoring with step-by-step analysis (tool choice, argument construction, result processing, synthesis), a "Loop" feature that generates a custom scorer from a natural-language criterion, and online scoring that runs an LLM judge on production traces with no ground truth.

TRACE (arXiv 2510.02837) is the reference-free anchor to know: reference-free, multi-dimensional (efficiency/hallucination/adaptivity), with an evidence bank that accumulates what the agent has established across steps so a step can be judged in the context of the whole trajectory rather than in isolation. The cost of going reference-free is the cost of every LLM judge: it is less deterministic and it inherits the judge's biases — position, verbosity, self-preference — so a reference-free trajectory score is only as trustworthy as its calibration (see judge calibration & meta-evaluation). The evals-101 warning — that a metric you have not validated is decoration — is doubly true when the metric is an LLM reading a trajectory.

State-based grading and the PRM crossover.

There is a third thing to check that neither the transcript nor the tool-call syntax captures: did the world actually change the way it was supposed to? tau-bench (Sierra, arXiv 2406.12045) grades by state. After the agent runs, it compares the resulting database state against an annotated goal state — it confirms an order actually flipped to "cancelled" in the DB, not merely that the agent said it did, and not merely that a syntactically valid cancel_order call was emitted.

tau-bench  —  grade by the resulting STATE, not by the transcript
--------------------------------------------------------------------
goal_state   orders["order_5567"].status == "cancelled"
db_after     orders["order_5567"].status == "cancelled"    -> MATCH
             (agent said "cancelled" AND the DB row actually flipped)

a transcript-only eval would be fooled by:
  agent answer   "your order is cancelled"
  db_after       orders["order_5567"].status == "open"     -> FAIL

This closes the gap the STEP 1 trace opened. An agent that says "cancelled" while the row still reads "open" passes outcome eval on the answer string and can even pass a tool-call-syntax check, but fails a state grader. τ²-bench (arXiv 2506.07982) extends this to dual-control settings where both the user and the agent call tools, which is closer to real support and operations work. State-based verification is the strongest process signal available when the environment has a checkable state; it is also the most expensive to set up, because someone has to annotate the goal state for every task.

The machinery for scoring individual steps was not invented for evaluation — it was built to train reasoning models with dense per-step reward, and it transfers directly. "Let's Verify Step by Step" (Lightman et al., OpenAI) showed process supervision beats outcome supervision and released PRM800K; Math-Shepherd (arXiv 2312.08935) and OmegaPRM (arXiv 2406.06592) replaced human step labels with automatic MCTS-style and divide-and-conquer Monte Carlo rollout labeling. A trained process reward model is a per-step scorer, and nothing stops you from pointing it at a trajectory as an evaluator — a learned or generative PRM scoring each step is an alternative to a single reference-free LLM-judge pass, and the same auto-rollout labeling that trains it can bootstrap trajectory annotations without exhaustive human labeling (see process reward models).

The pitfall crosses over with the machinery. A PRM trained on imperfect supervision can be reward-hacked, and a PRM used as an eval gate can be gamed exactly like a PRM used as a training signal — the agent, or the optimizer above it, learns to produce steps the PRM scores highly rather than steps that are actually good (see reward design & hacking). Whether you use the step scorer to train or to grade, the failure mode is identical, which is why process-versus-outcome is a live design axis rather than a solved question.

Pitfalls, and portable trajectories.

The first and most damaging pitfall is treating strict exact-match on the full trajectory as the default. Most real tasks admit many valid paths to the same answer — two independent tools can be called in either order, an optional confirmation step is legitimately optional — and exact-match fails every correct agent that took a different-but-valid route. The fixes are all in the tooling: use unordered/subset/superset matching instead of strict, allow multiple reference trajectories, or drop to reference-free judging. Reserve strict for the rare task with exactly one correct path.

The second pitfall is judging steps in isolation. A step that is locally fine — a reasonable tool call with grounded arguments — can be globally redundant or off-goal, and a judge that sees only one step at a time cannot tell. This is why TRACE's evidence bank matters: it carries what the agent has established across the whole trajectory so a step is scored in context, not in a vacuum. Any step-level judge blind to the surrounding sequence will miss global-coherence failures. Third, over-strict exact argument matching is brittle: two semantically-equivalent search queries, or two equivalent JSON encodings, are not byte-identical, and exact will fail them — reach for ignore or a field-list override in tool_args_match_overrides on the arguments where semantic equivalence is expected. Fourth, reference-free judges inherit the standard judge biases; nothing about reading a trajectory instead of a single answer immunizes them against position, verbosity, and self-preference bias.

All of this presupposes you can get the trajectory out of your agent in a shape the evaluator understands, and the thing that makes that vendor-neutral is the OpenTelemetry GenAI semantic conventions. They standardize the span/operation names — gen_ai.operation.name takes values invoke_agent, chat, execute_tool, create_agent — and the attributes: gen_ai.agent.name, gen_ai.tool.name, gen_ai.input.messages, gen_ai.output.messages, and gen_ai.usage.input_tokens / output_tokens. The conventions are still Development/experimental as of 2026, but they are already what lets Phoenix, LangSmith, and Braintrust ingest the same trajectory spans instead of each demanding a bespoke export. Emit OTel-shaped trajectories and your process evals stop being locked to one vendor's SDK.

Trajectory eval is not a replacement for outcome eval; it is the layer that tells you whether an outcome you trust was earned or lucky. Wire it in the way the rest of this group prescribes: as a CI gate with a living golden set rather than a one-off (see eval-driven development & CI), and read alongside the cost-and-reliability panel that decides deployment (see HAL & asynchronous agent eval). Score the answer to know the agent worked once. Score the trajectory to know it will work again.