Agent observability & tracing.
An agent that fails on step 14 of a 30-step run tells you exactly one thing — "it didn't work" — unless you instrumented it, in which case you can replay the exact prompt, the exact tool call, and the exact result that sent it wrong. This entry covers what to record, what to record it as, and the difference between watching a system and evaluating one.
A run is a tree, not a line.
The unit of observability for a normal web service is a request. The unit for an agent is a trace: one complete run, decomposed into nested spans that mirror the agent loop. A typical shape:
- The run — one user goal, from first input to final answer or give-up.
- Inside it, one step per iteration of the loop.
- Inside each step, a model call (what went in, what came back) and zero or more tool calls (arguments, result, error).
- Sub-agents, retries, and retrieval calls nest the same way.
Two properties of that tree are what make it useful. It preserves order — you can see that the agent searched before it wrote, or that it wrote first and searched to justify. And it preserves causation — each step's input contains the previous step's output, so you can walk backward from a bad action to the observation that produced it.
This is why the ordinary log line is not enough. A log tells you an event occurred; a trace tells you what the model was looking at when it decided.
The minimum viable trace.
Six things, and most teams discover the hard way that they are missing three of them:
- The prompt as actually sent. Not the template — the fully assembled text, after retrieval, memory, and history were spliced in. Almost every "why did it do that?" is answered by reading what the model actually saw, and a template cannot tell you that.
- The raw model response. Including tool calls with their exact arguments, and any reasoning the provider exposes. The arguments matter enormously — a wrong action is usually a right tool with a wrong parameter.
- The tool result. What came back, including errors and timeouts, exactly as the agent saw it. A truncated or reformatted result in the trace makes the next step's reasoning unreproducible.
- Cost and latency per span. Input tokens, output tokens, cached tokens, wall-clock. Aggregated over a run this gives you cost-per-task, which is the number anyone paying for the system will eventually ask for.
- A trace ID threaded everywhere. One identifier that appears on every span, every downstream service call, and every user-visible artifact — so a support ticket can be turned into a replay.
- Versions of everything that can change. Prompt version, model identifier, tool-schema version, retrieval index version. Without these, a regression is unattributable: you know behavior changed, but not which of four simultaneous changes did it.
Three questions, three uses.
Instrumentation earns its keep by answering questions at three different zoom levels:
- What happened in this run? Debugging a specific failure. You open the trace, find the first step where the trajectory diverged from what it should have been, and read the input to that step. Nearly always the answer is visible there — a retrieval that returned nothing, a tool error the model interpreted as a result, a stale instruction still sitting in the context.
- What happens across runs? Aggregates turn anecdotes into engineering: task success rate, cost and step count per task, tool error rate by tool, how often the agent hits its step limit, where the latency actually goes. A tool with a 30% error rate is a design problem you would never spot one trace at a time.
- Did this change help? Compare the same population of tasks before and after a prompt edit or a model swap. Without versioned traces you are relying on impressions, and impressions on a stochastic system are unreliable in both directions.
Observability and evaluation are complementary, not the same. Evaluation asks "how good is it on a fixed set of tasks I chose?" — offline, repeatable, with known answers. Observability asks "what is it doing on the real traffic I actually have?" — online, unlabeled, and full of inputs you never thought to write an eval for. The traffic is also your best source of new eval cases: the interesting production failures are exactly the tests you were missing.
How instrumentation goes wrong.
- Logging only the final answer. The single most common mistake. The final answer is where the failure surfaced; the trajectory is where it happened. A run can produce a plausible final answer through a completely broken path — and a run can fail at the last step after doing everything right.
- Recording the template instead of the render. Cheaper to store, useless to debug. If you only keep the template plus a reference to the inputs, you must be able to reconstruct the exact prompt byte-for-byte — and once retrieval and memory are involved, that reconstruction usually is not reproducible.
- Sampling away the failures. Sampling 1% of traces to control storage cost is reasonable; sampling uniformly is not. Keep every failed, escalated, expensive, or unusually long run, and sample the boring successes.
- Leaking secrets into the trace. A full-prompt trace contains everything the agent saw — customer data, retrieved documents, tool credentials that leaked into a shell command. Traces become a second copy of your most sensitive data, usually with weaker access control than the original. Redact at capture time, not at read time.
Instrument before you need it. Agent failures are frequently non-reproducible — a retrieval that returned different documents, a model that sampled a different branch, an external service that behaved differently that afternoon. If the trace was not captured at the time, the run is gone, and you are left debugging a description of a failure rather than the failure itself.
The wiki's tracing & observability chapter covers the production instrumentation stack, incident response for agents covers what to do with a trace once something has already gone wrong, and the trajectory & process evaluation deep-dive covers scoring the steps a trace records rather than only the answer at the end.