Synthetic data.
Synthetic data is model-generated training data, and the reason it went from a curiosity to standard practice is not that it is cheaper — it is that you can generate exactly the examples you lack, which no amount of scraping will give you. The failure mode is equally specific: a model trained on its own unfiltered output drifts toward its own average, so the entire craft is in the verification step, not the generation step.
The three jobs it actually does.
"Synthetic data" covers several distinct techniques that share a mechanism and little else:
- Distillation. A large model produces outputs; a smaller model trains on them, inheriting behaviour it could not have learned from raw text at its size. This is how most small instruction-following models are made — see distillation & quantization.
- Coverage filling. You have 40 real examples of a rare case and need 4,000. Generation targets the specific hole — the unusual claim type, the malformed input, the language you have little data in.
- Preference and reasoning data. Generate many candidate answers, score them, and train on the winners. This is the engine behind much of the reasoning-model improvement of recent years, and it works because verifying a solution is often far easier than producing one.
That last asymmetry is the load-bearing idea. Where a cheap, reliable checker exists — unit tests pass, the proof verifies, the SQL returns the right rows, the JSON matches the schema — you can generate at volume and keep only what passes. Where no such checker exists, you are training on unverified model output, and the quality ceiling is whatever produced it.
Model collapse, and what actually causes it.
The widely-cited result is that models trained recursively on their own output degrade — tails of the distribution vanish first, then the whole thing narrows toward the mean. It is real, and it is routinely over-generalised into "synthetic data is dangerous," which is not what the research shows.
Collapse is a consequence of a specific setup: generation with no filtering, no fresh real data, and no external signal, repeated across generations. Change any one of those and the picture changes:
- Filtering breaks the loop. If a verifier discards the bad samples, you are not reinforcing the model's errors — you are selecting against them.
- Mixing with real data anchors the distribution. Accumulating real and synthetic together behaves very differently from replacing real with synthetic.
- External signal adds information. A test suite, a compiler, a search result, or a human rating injects something the generator did not already know. Without that, generation can only redistribute existing knowledge, never add any.
The practical reading: synthetic data is safe in proportion to how good your filter is. It is not a property of the data, it is a property of the pipeline.
Where it earns its place in an agent stack.
Most teams building agents will never train a foundation model, and still have three good uses for generated data:
- Evaluation sets. The most under-rated use by a distance. You need adversarial cases, edge cases, and enough volume for a signal, and hand-writing them is where eval efforts die. Generate candidates, then have a human accept or reject — that inverts the expensive part from authoring to reviewing.
- Bootstrapping before you have traffic. A new agent has no production transcripts. Generated user requests, weighted toward the awkward ones, let you test the loop before real users find the bugs for you.
- Red-teaming. Generating prompt-injection attempts and jailbreak variants is cheap and scales past what a human team will patiently write.
Notice all three are about testing, not training. That is the honest default for application teams: the return on generated evaluation data is immediate, and the return on fine-tuning data only exists after you have exhausted prompting and retrieval.
The traps, in the order teams hit them.
- Homogeneity. Ask a model for 1,000 examples and you get one example with 1,000 paraphrases. Vary the seed material, not just the temperature — real inputs, personas, constraints, formats. Measure diversity explicitly (distinct n-grams, embedding spread) rather than eyeballing the first ten.
- Leakage into evaluation. If the same model generates both your training data and your test set, your benchmark is measuring self-consistency. Hold out real data for evaluation, always.
- Inherited bias, concentrated. Generation reproduces the generator's skew and filtering can sharpen it, because the filter shares the same blind spots.
- Licence and provenance. Output from a hosted model carries terms, and some prohibit training a competing model on it. This is a contract question before it is a technical one — check before you build a pipeline on it.
- Privacy is not automatic. "Synthetic" does not mean "anonymous". A generator conditioned on real records can reproduce them; if that matters, you need a formal guarantee, not a vibe.
Before generating anything, write down the checker. If you cannot state how a generated example will be verified — a test that runs, a schema that validates, a rule that fires, a human who reviews — you are not building a synthetic data pipeline, you are building a paraphrase machine that will confidently teach a model its own habits. The checker is the product; generation is the cheap part.
Related: training vs inference for where this sits, agent evaluation for the use with the best return, and reading benchmarks for why contaminated test sets flatter everyone.