Automatic Prompt Optimization

The prompt is the last unfitted parameter in the system.

Every other numeric part of a model pipeline is fitted against data. Weights are trained. Retrieval thresholds get tuned on a validation set. Only the prompt — the component with the largest effect on output quality in most agent systems — is written by a person, judged by reading it, and shipped because it seemed good in a handful of manual tries.

• Human prompt-tuning has a sample size of about three. You try a wording, run two or three inputs, read the outputs, and form a belief. Two or three samples cannot distinguish a real five-point improvement from noise; the statistics of eval variance are unforgiving on this point, and they apply to your informal testing exactly as they apply to a formal suite.
• Readability and score are only loosely related. Optimizers routinely land on instructions that a human editor would cut — redundant restatements, oddly specific formatting demands, examples that look unrepresentative. They win on the metric anyway, because the model is not reading for style.
• The search space is bigger than the wording. Which examples you include, how many, in what order, how the output is structured, whether the task is one call or three — all of it is searchable, and the biggest wins usually come from the example set rather than the instruction text. That is the same lever few-shot prompting describes, now selected by measurement instead of by taste.

How the optimizers actually work.

The published methods differ in what they mutate and how they choose the next candidate, but every one of them is the same loop: propose a candidate prompt, score it on a dataset, keep what wins, repeat. Three families cover most of what ships.

• Example selection and bootstrapping. Run the current prompt over your training inputs, keep the traces that scored well, and promote them into the prompt as demonstrations. Cheap, robust, and frequently the single largest gain — the model learns your task's shape from its own successful attempts rather than from your hand-written examples.
• Instruction search. A second model proposes rewrites of the instruction text, each is scored, and a search strategy — random, evolutionary, or Bayesian over a proposal space — decides where to look next. This is where the surprising, ugly-but-effective wordings come from.
• Reflective, feedback-driven mutation. Instead of scoring a candidate with a single number, feed the optimizer the failures in natural language — the wrong outputs, the judge's stated reasons, the tool errors — and let it write a targeted revision. This is the direction the DSPy/GEPA line of work took, and it is dramatically more sample-efficient than blind search because each rollout carries diagnostic information rather than one scalar. See DSPy 3 and GEPA for agent optimization for how this composes across a multi-stage pipeline.

For a multi-step agent the important property is that these compose over a program, not a string. If your pipeline has a classifier stage, a retrieval stage and a writer stage, the optimizer can fit all three against one end-to-end metric — which is the only way to discover that the writer's problem was the classifier's phrasing.

The metric and the held-out set are the whole job.

An optimizer will maximise exactly what you asked for, on exactly the data you gave it. Both halves of that sentence are traps, and both are the reason teams report that automatic optimization "didn't work" — the loop worked fine; it optimized something they did not want.

• Without a held-out split you are not optimizing, you are memorising. Search over a few dozen candidates on fifty examples will find a prompt that fits those fifty examples' quirks. Split before you start, report only held-out numbers, and keep a third set you touch once, at the end.
• A weak metric gets gamed faster than a weak prompt. If the metric is an LLM judge, the search will find the prompt that flatters that judge — longer answers, confident framing, the format the judge was trained to like. Calibrate the judge against human labels before you let an optimizer run against it, or you are fitting to a bias.
• Fifty to two hundred well-labelled examples beats two thousand scraped ones. Label quality caps everything downstream; a search that is being scored by noisy labels converges on noise. This is the same ceiling described in annotation and labeling ops.
• Score the outcome you are paid for, and include cost. An optimizer with a pure quality objective will happily discover that adding six more examples and a longer instruction improves accuracy by one point — and quadruples your per-call input tokens forever, on every step of every run. Put tokens in the objective or cap them as a constraint; agent cost control explains why that inflation compounds in a loop.

The optimized prompt is a build artifact with an expiry date.

This is the part that surprises teams after the first successful run. You now hold a machine-generated prompt that nobody on the team wrote, that contains demonstrations selected by a search, and that was fitted to one specific model version. It is a compiled output, and it needs the handling a compiled output gets.

• Version it with the model ID, the dataset hash and the score. A prompt whose provenance is "the optimizer produced it in March" cannot be debugged or reproduced. Store the artifact alongside the inputs that made it, the way reproducibility requires of any generated component.
• Re-fit on every model change, and budget for it. The prompt was fitted to one model's quirks; a new checkpoint has different ones. Re-running the optimizer is cheap compared with the first run — the dataset already exists — but it is a step in your migration checklist, not an optional cleanup.
• Keep the hand-written prompt as the readable spec. Humans still need to know what the system is supposed to do. Maintain the intent in prose, let the optimizer produce the shipped string from it, and never edit the generated artifact by hand — an edited artifact is unfitted again, with none of the honesty of an unfitted one.
• Do not optimize what you cannot yet measure. If you have no scored dataset, the correct first move is to build one, not to install an optimizer. That dataset pays for itself against regressions and model swaps regardless of whether you ever run a search — which is the argument in eval-driven agent development.

Related: prompting basics for the levers being searched, context engineering for everything else that fills the window, evals 101 for the scored set this all depends on, and the agent harness for why the prompt is only one of several things a benchmark number is secretly measuring.