Training on Your Own Traces

Step 1Three filters sit between production and your dataset, and nobody chose any of them.

A trace is not a sample of "tasks an agent should do well." It is a sample of tasks that survived three filters, each applied by a system with its own agenda.

What users attempted. Users learn your agent fast. Within a month they have stopped asking for the things it does badly, which means the distribution you are logging is the distribution your current model is already good at. The hardest category in your product is also the rarest category in your logs, and it is rare because it is hard.

What completed. Runs that hit a step limit, a timeout, a tool 500 or a refusal are over-represented in your error dashboards and under-represented in anything a training pipeline would call usable, because they have no clean final answer to supervise against. The trajectories that look most like training data are the ones where nothing interesting happened.

What you kept. Your sampling and retention policy is now a data-selection policy. Head-based sampling at 5% silently discards 95% of the tail you most want; a 30-day window deletes exactly the seasonal cases you cannot generate; a zero-retention commitment to one enterprise customer removes your most demanding workload from the corpus entirely.

Step 2The quadratic term: why 99% per-step accuracy is not a 26% failure rate.

Here is the arithmetic every flywheel proposal runs, and the one it is missing. Suppose your fine-tuned policy deviates from the demonstrated action on 1% of steps. Over a 30-step task, the independent-errors calculation gives 1 − 0.99³⁰ = 0.26 — a 26% chance of at least one deviation, which sounds survivable, and teams ship on it.

It is the optimistic bound, not the realistic one. The classic imitation-learning result — restated for LLM agents in a May 2026 paper revisiting DAgger — is that behaviour-cloning error does not accumulate linearly in the horizon T; it accumulates quadratically, because the states are not independent. The first deviation moves the agent into a state distribution the demonstrations never covered, where its error rate is not 1% but unknown, and every subsequent step compounds from there. Training on states induced by the teacher (your old policy, or a stronger model you distilled from) while evaluating on states induced by the student is the train–test mismatch the whole field is named after.

Two consequences follow, and both are practical. First, the horizon is a first-class term in your dataset design: the same corpus that produces a solid 5-step tool-calling model produces a disappointing 40-step research agent, and the fix is not more of the same data. Second, the data you most need is data about recovery — what a good policy does from a state a bad policy created — and that is precisely the data a success-filtered trace log does not contain. The DAgger remedy is to shift the state distribution gradually from teacher-induced toward student-induced: run the candidate policy, collect the states it reaches, and label the right action there.

# the two datasets are not the same shape
teacher_states = traces.filter(success=True)       # where the OLD policy went
student_states = rollouts(candidate).collect()  # where the NEW policy goes

# round k: label the states the student actually reaches
for k in range(rounds):
    s = mix(teacher_states, student_states, beta=0.9 ** k)
    labels = oracle(s)        # verifier, stronger model, or human
    candidate = train(candidate, s, labels)
    student_states = rollouts(candidate).collect()

Note what the loop requires that a trace log does not supply: an oracle for states nobody has seen yet. If you cannot produce one — a verifier, a test suite, a stronger model, a human in a labelling queue — the flywheel does not turn, and environment engineering is the prerequisite project rather than a later one.

Step 3Rejection sampling discards exactly the distribution you need.

The standard recipe is rejection fine-tuning: sample trajectories, keep the ones the verifier marks correct, train on those. It works, it is cheap, and it is the first thing to try — Nebius's public OpenHands work released roughly 67k trajectories together with RFT checkpoints and reported that it captures much of the quality of a reinforcement-learning run without the rollout infrastructure an RL loop demands. That is a real result and the right starting point; SFT, rejection sampling & distillation covers the mechanics.

What the filter costs is worth naming precisely. Keeping only correct trajectories removes the full distribution of attempts, which means the tuned model never observes the failure modes it will meet in production — and in an agent loop, observing a failure and recovering is a skill with its own token budget. A model trained exclusively on runs where the first tool call returned 200 has no representation of what to do with a 429, a partial result, or a tool that answered confidently and wrongly.

So change the label, not the filter. A trace has at least four outcomes worth distinguishing — succeeded cleanly, succeeded after recovering from an error, failed on the environment, failed on the model — and the second category is the valuable one. Keep failures with their outcome labels and use them for evaluation and for negative-preference signal rather than discarding them at ingest, where the decision becomes irreversible. Pushing the rejection-sampling or preference rounds too hard has its own cost on the other side: diversity collapse, where the model converges on one style of solution and loses the ability to try a second approach when the first does not work.

Step 4The harness is in the data, which makes the fine-tune a dependency on your own tool surface.

This is the cost that surprises people after the model ships. A trajectory is not abstract reasoning; it is a sequence of tool names, JSON argument shapes, system-prompt phrasing and error strings, all of which are properties of your harness on the day the trace was written. Fine-tune on that and you have encoded your tool catalogue into the weights.

Then ordinary maintenance becomes a model problem. You rename search_docs to search_knowledge_base; you tighten a schema from a free-text filter to an enum; you split one tool into two because the first was too coarse. Each of those is a routine improvement that a prompted model absorbs from its tool definitions in a single turn, and that a trace-tuned model fights, because its prior now favours a call shape that no longer exists. The team's instinct is to freeze the tool surface to protect the fine-tune, which is the moment the asset becomes a liability — you have bought a few points of accuracy with your ability to change the product.

There is a second, quieter version in evaluation: traces used for training and traces used for scoring come from the same pipe, so your eval set is drawn from the same distribution your weights memorised, and improvements measured against it are partly contamination. Split the corpus by time and by customer before anything touches a loader, and hold the newest slice out entirely.

Step 5What you are allowed to keep is a design constraint, not a legal footnote.

Production traces are the richest personal-data artifact most agent teams own: the user's actual request, the documents retrieved on their behalf, the arguments passed to a CRM, sometimes the contents of an inbox. Two obligations follow and they pull against each other. Redaction protects the subject and degrades the data — a trace with the entity names replaced by placeholders is still useful for measuring tool-call structure and nearly useless for teaching retrieval judgement. Meanwhile a deletion request has to reach the corpus, and once a trace has been baked into weights there is no row to delete; the only honest answer is a retrain, which is why data governance for an agent has to decide in advance which corpora are trainable at all.

The practical shape most teams land on: a short-retention, un-redacted operational store for debugging and replay, and a separate, consent-scoped, redaction-processed corpus that is the only thing a training job may read. Two stores, two retention clocks, one of them nameable in a contract. Decide this before the flywheel ships, because retrofitting consent onto a corpus you have already trained on is the one failure in this essay with no engineering remedy.

Step 6The order of operations that actually pays.

Traces are genuinely valuable; they are just valuable in an order most teams get backwards. Run it in this sequence and each step pays for the next.

First, the eval set. Mine failures — not successes — for the cases your agent gets wrong, cluster them with a failure taxonomy, and turn each cluster into a scored case. This is the highest-value use of a trace log by a wide margin and it needs no GPU. Second, the environment. Replay the hard traces into a sandbox with a verifier, which converts a transcript into something that can be re-run — the asset that survives the model you are about to train, as environment engineering argues. Third, the prompt and the tools. Most clusters in a real taxonomy dissolve into a tool whose error message was unreadable or a schema that invited the wrong call; those fixes ship today and cost nothing in flexibility. Fourth, and only now, weights. Start with rejection fine-tuning on the verifier you built in step two, then add student-induced states if the horizon is long.