Measuring Agent Latency

The unit is the trajectory. The call is a component.

Almost every latency dashboard in an agent stack was inherited from an API mindset: requests in, milliseconds out, percentiles per endpoint. That instrument is correct for a single completion and structurally misleading for an agent, because the user is not waiting for a call. They are waiting for a task, and the task is a variable-length chain whose length is itself a random variable.

• Report duration per completed task, keyed to whatever the user asked for, with step count recorded alongside it. Two runs of the same task at 40 seconds — one at four steps, one at nineteen — are different systems wearing the same number.
• Step count is a latency metric, not a quality metric. It is usually the largest single driver of trajectory duration and the one most sensitive to a prompt change, which makes it the first thing to chart when a p95 moves without an infrastructure change.
• Failed and capped runs belong in the distribution. Excluding them flatters the number precisely where users are suffering most: a run that hits the step cap at 300 seconds and returns nothing is the worst latency event in your system, and dropping it as an error makes your latency chart improve as your product degrades.
• Retries are part of the duration. If the orchestration silently retries a failed tool call three times, the user waited for four attempts. Log the total, not the successful attempt — see idempotency & retries.

Chaining makes the tail the median.

This is the arithmetic that justifies the whole page, and it is worth doing explicitly because the intuition runs the wrong way. If a single step has a 1% chance of landing in its slow tail, the probability that a fifteen-step trajectory contains at least one such step is 1 − 0.99¹⁵ ≈ 14%. A one-in-a-hundred event at the call level is a one-in-seven event at the task level.

• Optimising the median step barely moves the task. Shaving 15% off a p50 that is already fast changes the sum slightly; removing the p99 spike changes what a seventh of your users experience. The tail is where the budget goes.
• On shared serverless capacity, your tail is other tenants. That is a large and often unrecognised argument for dedicated capacity — the predictability is the product, and cost is the justification written afterwards. See provisioned throughput & commitments.
• Long-horizon agents are the extreme case. At forty steps a 1% per-step tail becomes a 33% chance per task. Trajectory length and tail latency multiply, which is why capping steps is a latency control as much as a cost control.
• Percentiles do not add. You cannot sum per-step p95s to get a trajectory p95. Measure the trajectory end-to-end and let the distribution tell you, rather than assembling it from component percentiles.

Decompose the trajectory before you blame the model.

Teams reliably assume the model is the slow part, and reliably find something else when they finally instrument it. A trajectory's wall-clock splits into four buckets that require different fixes and different owners.

• Model time — time-to-first-token plus generation. Sensitive to context length, so it grows as the trajectory grows; a step-twelve call is slower than a step-two call on the same model for reasons that have nothing to do with the provider.
• Tool time — the external systems the agent calls. Frequently the largest bucket and almost always the one with the worst tail, because it inherits the latency profile of whatever legacy API is behind it.
• Queue and scheduling time — rate-limit backoff, concurrency caps, cold starts, waiting for a sandbox. Invisible in a naive trace because nothing is executing, and often the dominant term under load — rate limits & provider capacity.
• Your own orchestration — serialisation, retrieval, guardrail passes, context assembly. Small per step and multiplied by step count, which is how a 200 ms retrieval call becomes eight seconds of a trajectory.

Record all four as separate spans with a shared trajectory ID, and the attribution question answers itself. That is the structural argument in tracing & observability, and the span kinds are already standardised — use them rather than inventing labels, per OpenTelemetry GenAI semantic conventions.

Time-to-first-useful-output is a different SLO from time-to-done.

The two questions "when did something appear" and "when was the task finished" have different answers, different owners and different users, and collapsing them into one metric is the most common measurement mistake in this area.

• For an interactive agent, the felt latency is time-to-first-useful-output. Not the first token — a spinner replaced by a token is not progress. The first moment the user learns something they did not already know: a plan, a found file, a partial answer.
• For a background agent, time-to-first-output is nearly irrelevant and completion time is everything. Instrumenting both the same way sends you optimising streaming for a workload nobody is watching — the distinction drawn in background coding agents.
• Perceived duration is not measured duration. A visible, honest progress signal changes tolerance substantially, which means some latency work is interface work rather than infrastructure work — waiting & latency UX.
• Beware the metric that improves when you stream more and finish later. If you only chart first-output, you will reward exactly that trade. Chart both, always, side by side.

Record enough per step that the number is reconstructable later.

A duration with no context is a number you cannot act on six weeks later, when the p95 has moved and nobody remembers what changed. The fields below cost almost nothing to emit and are the difference between diagnosing a regression and guessing at it.

• Model ID and version, per call. Providers move models under stable aliases; a latency shift with no deploy on your side is usually this. Pin and record, per model deprecation & migration.
• Input token count and cached-token count. Time-to-first-token tracks prompt length closely, and a drop in cache hit rate looks exactly like a provider slowdown while being entirely your routing. This is the operational half of prompt caching.
• Step index and termination reason. Which step, out of how many, and why the run stopped — answered, gave up, hit the cap, errored. Termination mix is a leading indicator for both latency and quality.
• Queue wait as its own span. If it is folded into the model call, you will spend a quarter on the wrong vendor conversation.
• Keep the slow runs whole. Naive head sampling discards the tail you are trying to study; buffer to run end and retain every slow, capped and failed trajectory — trace sampling & retention.

Write an SLO that survives a model change.

Latency for an agent is not a fixed property of your code; it moves when a provider re-routes, when a prompt grows, when a tool's owner ships a slower query. An SLO that assumes stability will be permanently in breach or permanently meaningless.

• Set the objective on the trajectory, at a percentile, per task class. "95% of support-triage tasks complete within 45 seconds" is actionable. "p50 model latency under 800 ms" is a component target masquerading as a user promise.
• Separate the classes. A one-tool lookup and a twelve-step research task do not belong in one distribution; averaging them produces a number that describes neither and hides regressions in both.
• Give latency an explicit budget in the eval suite. A prompt change that adds three steps for half a point of quality is a regression, and it will pass any eval that only scores correctness — eval-driven agent development.
• Define the degraded mode before you need it. When the budget is blown, what gets cut — a smaller model, fewer retrieval passes, a shorter step cap? Decide once, in advance, per graceful degradation & fallback.
• Alert on step-count and termination-mix shifts, not only on duration. Both move before the duration percentile does, which buys you the only useful thing an alert can buy: time.

Related: SLOs & error budgets for the surrounding practice, concurrency & scaling for what happens to these numbers under load, and cost, quality, latency for the three-way trade this sits inside.