Trace Sampling & Retention

Agent traces do not fit the logging mental model, and the arithmetic is why.

Teams size their trace budget from experience with request logs and are wrong by one to two orders of magnitude, because a trace is not a line per event — it is the whole context, repeatedly.

• The payload grows with the square of the step count. Models are stateless, so step twenty re-sends everything from steps one to nineteen. If you record the prompt and completion at each step — and you must, or the trace cannot be replayed — a twenty-step run stores something closer to two hundred step-equivalents of text. The same quadratic that drives agent cost drives the storage bill.
• Payload dwarfs structure by a factor nobody guesses correctly. The span skeleton — names, timings, status, token counts, tool names — is kilobytes per run. The prompts, retrieved documents and completions are megabytes. They are stored together by default, and they have completely different value curves.
• Retrieval makes it worse. An agent that pulls twenty chunks per step and records what it retrieved — necessary to debug a wrong answer — multiplies the payload again, and those chunks are duplicated across every run that retrieved them.
• So you will sample. Not as a design preference but as an arithmetic consequence. The only real question is which runs survive, and that question has a clearly right answer that most systems get wrong by default.

The default is head-based, and it is anti-correlated with what you want.

Head-based sampling decides at trace start, before anything has happened. It is the default in most instrumentation because it is the only thing you can decide cheaply and locally — and it throws away information in precisely the wrong direction.

• Failures are rare, so uniform sampling makes them rarer. A control that keeps 1 in 10 runs keeps 1 in 10 incidents. The population you actually study — bad trajectories — shrinks by the same factor as the population you are drowning in, which is the opposite of the intended effect.
• The tail is where the money and the danger are. The p99 run costs orders of magnitude more than the median and is usually a loop that never terminated. Uniform sampling keeps 1% of them, and the runaway you most needed to see is the one you did not keep.
• Nothing at trace start predicts the outcome. That is the structural point. In a conventional service, the endpoint and the customer tier tell you a lot about a request before it runs. In an agent, two runs with identical inputs diverge at step four, and the interesting one is not identifiable at step zero.
• Ad-hoc "debug mode" sampling does not rescue it. Turning full tracing on for a tenant after they complain gives you future runs, not the one they are complaining about — and the complaint is usually about a run from last week.

Decide at the end: buffer the run, then keep it on outcome.

Tail-based sampling holds the trace until the run completes and then decides, which costs you a buffer and buys you the ability to condition on what happened. For agents this is not an optimisation, it is the difference between a usable archive and an expensive one.

• Keep 100% of anything that went wrong. Errors, timeouts, step-cap terminations, guardrail refusals, tool-call failure loops, human overrides and rejections, low-rated sessions. This population is small by definition, so keeping all of it is affordable — and it is the whole point of the archive.
• The outcome signal is the shape of the run, not a label. Production has no ground truth, so define "interesting" from what you can observe without a judge: hit the step cap, retried the same tool three times, terminated without producing the expected artifact, cost above the p95, latency above the p95, or the user re-asked immediately. These are the same indicators quality regression detection builds on, and they are cheap to compute at run end.
• Keep a stratified baseline of successes, not a uniform one. You need normal runs for comparison, but not proportionally — sample successes by task type, tenant and model version so that rare task classes stay represented. Uniform success sampling gives you ten thousand copies of your most common workflow and nothing from the workflow that only runs at month end.
• Always keep the skeleton for every run, sampled or not. Metrics computed from a sample have sampling error; metrics computed from every run's span structure do not. This split is the reason the next step matters.
• Budget for the buffer. A long-running agent may hold a trace in memory for minutes. Cap the buffer, spill to disk, and decide what happens when it overflows — usually keep the skeleton, drop the payload, and record that you did.

Split the trace in two at the collector, with two retentions.

Structure and payload have different value curves, different privacy exposure and different costs, and storing them in one place forces a single retention policy that is wrong for both.

• Structural telemetry is cheap and stays useful for years. Step counts, tool names, durations, token counts, model versions, outcome codes. This is what trend analysis, capacity planning and "did the p95 step count change after the model swap?" run on. Keep it long — it is small enough that the argument for deleting it is weak.
• Payload is expensive and its debugging value decays in weeks. Nobody debugs a specific trajectory from four months ago. Its dataset value, however, does not decay at all — which is the tension the next step is about.
• Split at the collector, not at the vendor. One collector hop that forks structural spans to your metrics store and payloads to storage you own is the change that makes every later decision reversible, which is the argument the OpenTelemetry GenAI conventions page makes for instrumentation generally. It is also where redaction belongs.
• Redact on the way in, because you cannot redact retroactively at scale. Secrets, credentials and personal data in prompts are permanent once written to an append-only store. Do the detection at the collector, keep a hash so you can still tell two runs apart, and accept that some fidelity is the price of being able to keep the archive at all.

Retention is a dataset decision, and it collides with your deletion duties.

This is the part that gets decided by accident. Someone sets a 30-day retention to control cost, and eighteen months later the team discovers that the golden trajectories, the regression baseline and the only record of how the system behaved before the last three model migrations are all gone.

• Your trace archive is the eval set you have not built yet. Real production trajectories are the highest-value input to eval-driven development, and they cannot be regenerated — the model version, the index and the traffic that produced them are gone. Deleting them is not a storage decision, it is deleting the only training and evaluation data that reflects your actual users.
• Keep a permanent, curated slice deliberately. A few thousand traces, chosen for coverage rather than recency, promoted out of the sampling pipeline into a dataset with its own lifecycle. Once promoted, a trace is no longer telemetry — it is a fixture, and it should survive any retention sweep.
• Reconcile that against erasure and legal hold before you promote anything. Traces contain user content, so a deletion request must reach the curated set too — which means the promotion step has to preserve the subject linkage, or you have built an undeletable corpus. Conversely, a legal hold must suspend the sampler's deletion, and a sampler that has already dropped the run cannot comply. Both directions are worked through in retention and legal hold.
• Decide the retention with the eval owner in the room. The person who will need the data in a year is not the person optimising the observability invoice, and if only one of them is in the meeting the answer is predictable.

Operate it: measure what you dropped, and prove you can still replay.

A sampling policy fails silently. Nothing alerts when the archive stops being representative; the dashboards keep rendering and the numbers keep looking fine.

• Record the sampling decision and its reason on every run. Kept-as-failure, kept-as-baseline, dropped, buffer-overflow. Without this you cannot reweight a metric computed from the archive, and every number derived from it is quietly biased toward failures.
• Alert on the composition, not the volume. The signal is the ratio — if kept-as-failure traces jump from 2% to 9% of the archive, something regressed; if they fall to near zero, either the system improved or the detector broke, and those two need different responses.
• Drill the replay. Once a quarter, take a trace from the oldest tier you claim to keep and actually reconstruct the run from it. Missing tool schemas, an unrecoverable model version, redaction that removed something load-bearing — these are discovered during an incident otherwise, which is the worst possible time.
• Price the archive per completed task, not per gigabyte. Observability spend that is a few percent of inference spend is not worth optimising; a trace bill approaching the model bill means you are storing payloads you should be sampling, and that comparison is more useful than any absolute threshold.
• Never sample the evaluation environment. Eval and red-team runs are low-volume and maximally informative. Keep them whole, keep them forever, and keep them separate from production sampling — the cost is negligible and the alternative is an eval you cannot audit.

Related: tracing and observability for what a trace should contain, production feedback signals for routing runs into the eval set, the cost of evaluation for the other half of this bill, and audit trails and provenance for the records sampling must never touch.