Distillation & Quantization

Quantization: fewer bits per number.

A weight trained at 16 bits does not need 16 bits to be useful. Quantization maps those values onto a smaller numeric type — 8-bit, 4-bit, sometimes lower — and the payoff is close to linear in memory:

• Roughly 0.6 GB per billion parameters at 4-bit, against about 2 GB at 16-bit. That is what turns an 8B model into a 5 GB file that fits on a laptop.
• It usually makes generation faster, because inference on a single request is memory-bandwidth-bound. Moving fewer bytes per token is the speedup — not arithmetic.
• The KV cache is separate. Weights are the one-off cost; the cache grows with context length and concurrency, and long contexts are frequently what actually exhausts the GPU.

Quality loss is small down to about 4-bit and then stops being subtle. Modern methods do not round naively — they calibrate on sample data and protect the small number of outlier weights that matter disproportionately, which is why a good 4-bit quantization beats a naive 8-bit one.

Distillation: a smaller model taught by a bigger one.

Here the student is a genuinely different, smaller network trained to reproduce the teacher's behaviour. The reason it beats training that small model from scratch is that the teacher's outputs carry more information than a raw dataset does — the full probability distribution, or a worked chain of reasoning, rather than just the single correct answer.

• Response distillation — train on the teacher's outputs. Simple, effective, and what most "distilled" open models actually are.
• Reasoning distillation — train on the teacher's intermediate steps, which is how small models acquire behaviour that normally requires far more scale. It is also how a slow, expensive reasoning model gets compressed into something fast enough to serve.
• Task distillation — the one application teams actually do: use a frontier model to label or solve a narrow task, then train a small model on that. On a sufficiently narrow task the small model can match the teacher, because it only has to be good at one thing.

The cost is that distillation is a training run, with data preparation, hyperparameters, evaluation, and the licensing question that governs whether you may train on a hosted model's output at all — see synthetic data, which is the same pipeline viewed from the data side.

Choosing between them.

They are not alternatives so much as different answers to different constraints, and they compose — most local models you download are a distilled model that has then been quantized.

• Model is too big for your hardware → quantize. Minutes of work, no training, and reversible: you still have the original weights.
• Model is capable enough but too slow or costly at volume, on a narrow task → distil. Weeks of work, and only worth it if the task is stable and the volume is real.
• You need general capability → neither. Both trade breadth for economy. If the job genuinely needs frontier reasoning, buy frontier reasoning.

Two related techniques often get filed here and should not be. Pruning removes weights or whole structures rather than reducing their precision. Fine-tuning adapts a model's behaviour without changing its size at all — a different goal, covered in fine-tuning vs RAG vs prompting.

What breaks, and what to measure.

Both techniques degrade capability unevenly, which is exactly what makes them dangerous to evaluate casually — the average holds up while something specific quietly stops working:

• Long-horizon agentic work fails first. A one-point drop per step compounds across a twenty-step loop into a much larger drop per task. Quantized and distilled models look fine in single-turn chat and fall apart as agents.
• Rare knowledge and long-tail languages go before common ones. Compression hits the thin parts of the distribution hardest — the same shape of loss as model collapse.
• Structured output and instruction-following get brittle. Schema adherence and refusal behaviour are common casualties, and both are load-bearing in an agent.
• Safety behaviour is not guaranteed to survive. Alignment learned by the teacher is not automatically inherited by the student, nor preserved by aggressive quantization. If you rely on it, re-test it.

So do not evaluate on a benchmark average. Run your own task evaluation before and after, at the step count you actually use, and look at the tail rather than the mean.

Related: small & local models for running the results, open vs closed models for whether you may do this at all, and agent cost control for the cheaper levers to try before either.