Performance-Optimization Agents

Why this domain breaks the standard loop.

The generate-patch-then-run-tests loop in patch generation works because the oracle is binary, deterministic and cheap. Replace it with a benchmark and all three properties go: the oracle is a continuous measurement with variance, the variance depends on the machine and its neighbours, and each read costs seconds to minutes.

• The noise floor sets the floor on ambition. If your benchmark's run-to-run coefficient of variation is 5%, a single-run "8% faster" is not evidence of anything. The agent cannot tell the difference and, unlike a human, will not develop a superstition about it — it will simply accept and move on.
• Candidate breadth turns noise into false wins. This is the multiple-comparisons problem wearing a hoodie. Generate twenty patches that all do nothing, measure each once, keep the best: the winner will look like a comfortable improvement every single time. Breadth is the agent's main advantage over a human, and it is precisely what makes an unguarded loop dangerous.
• The benchmarks themselves have this problem. A 2026 reliability study of the main suites found SWE-Perf especially fragile because many of its own reference patches produce close-to-zero runtime change — the human expert's fix is inside the noise. If a published benchmark can ship that, your ad-hoc timing script can too.

The published numbers say the same thing from the other direction. Across GSO (102 tasks), SWE-Perf (140) and SWE-fficiency (498), models average well under a quarter of the expert speedup, routinely introduce correctness bugs in the process, and show a consistent preference for superficial edits over the structural change the expert made. None of that is a prompting problem.

Build the harness before you build the agent.

The agent's most important tool is one function, and its signature carries the entire discipline:

measure(workload, repetitions) -> {
  median_ms, ci95_low, ci95_high, n, machine_id, resolvable_effect
}

Three design choices in that return value do most of the work:

• Return an interval, never a scalar. Hand the model a single number and you have taught it that 412ms is faster than 418ms. Hand it 414 [396, 433] against a baseline of 419 [402, 440] and a competent model will say so itself. This is the highest-leverage line in the whole playbook and it is a formatting decision.
• Publish the resolvable effect up front. Compute, from the baseline's variance and the repetition budget, the smallest speedup this harness can actually detect, and put it in the agent's system prompt as a hard floor: changes below it are not accepted, no matter what the run said. See eval variance & statistical power — the arithmetic is the same one you use for eval sets, applied to milliseconds.
• Make the environment part of the identity. Pin CPU affinity, disable turbo where you can, fix the input data, separate warm from cold runs, and refuse to compare measurements taken on different machine_ids. Run it in the same isolated sandbox the patches run in — sandboxing & safe execution — because a noisy neighbour is indistinguishable from a regression.

Correctness is a gate, not part of the objective.

"Faster and wrong" is the dominant failure of every published system in this space, and it is structural: almost every real speedup removes work, and the model cannot always tell which of that work someone depended on. Never let the speedup and the correctness signal be combined into one score the agent optimises — make correctness a precondition that runs first and disqualifies silently.

• Full suite, not the fast subset. The pass-to-pass framing that SWE-fficiency uses is the right one: every test that was green before must still be green, and the agent does not get to choose which ones to run.
• Differential output checks for anything numeric. A tolerance-bounded comparison of outputs before and after, on inputs the test suite does not cover. Vectorising a loop and quietly changing accumulation order is a real speedup and a real behaviour change.
• Explicit invariant checks for the caching family. Memoisation, precomputation and lazy loading are the patches models reach for first, and they break in the two places tests rarely look: unbounded growth, and correctness under concurrent or repeated invocation. Ask for the eviction policy and the thread-safety argument in the patch description, and reject the patch if there isn't one.
• Two workloads, always. Optimising for the benchmark input at the expense of every other input is a legitimate patch under a single-workload oracle. A second input at a different size or shape costs one more measurement and kills the entire class.

Give the agent a profile, not a repository.

The largest reported gains in this domain come from telling the agent where the time goes rather than from a better model. PerfAgent — a profiler-guided iterative-refinement loop — roughly doubled the rate of expert-matching patches over a general coding-agent baseline on GSO (19.6% → 39.2%) and lifted SWE-fficiency-Lite from 26% to 74%. The mechanism is not subtle: without a profile, the agent optimises the code it can most easily read, which is uncorrelated with the code that runs.

• Attach the profile as context, not as a tool the agent may forget to call. Top-N functions by self time, the call tree above each, and allocation counts if the runtime gives them. This is the perf-specific answer to repo navigation & code context.
• Require a stated mechanism before the patch. "This function is 61% of self time; it re-parses the config on every call; hoisting the parse should remove that fraction." A patch with no hypothesis is a lottery ticket, and rejecting those cheaply is worth more than any prompt improvement.
• Check the mechanism against the measurement afterwards. The claim predicted a fraction; the harness measured one. When they disagree the patch is suspect even if it looks like a win — that gap is where you catch the accidental cache and the skipped work.
• Re-profile after every accepted patch. The hot path moves. An agent working from the original flame graph three patches later is optimising a bottleneck that no longer exists.

The patches that look like wins and are not.

Keep an explicit rejection list in the system prompt and, better, as automated checks in the harness. These recur across models and codebases:

• Under-floor micro-optimisation. Swapping a comprehension for a generator in a function that is 0.4% of runtime. Costs review time, buys measurement noise, and clutters the diff that a human has to approve.
• Silent resource trades. Faster because it now holds the whole dataset in memory. Measure peak RSS alongside wall-clock and treat a large regression as a failed gate, or you will find out in production.
• Work removed rather than accelerated. A validation pass deleted, a log line dropped, an assertion disabled. Fastest of all, and the reason correctness gates run on the full suite.
• Benchmark-shaped changes. A branch that special-cases the exact size or type the workload uses. The second workload from step 3 catches most; a human reading the diff catches the rest.
• Stacked patches measured once. Five accepted changes, each individually inside the noise, presented as a 12% total. Re-measure the stack against the original baseline with a full repetition budget before anyone believes the sum.

Ship the measurement with the patch.

The output of this agent is not a diff. It is a diff plus the evidence a reviewer needs to not re-run the experiment themselves, and the review is where the remaining false positives die.

• Put the numbers in the pull request: workload, baseline and patched intervals, repetition count, machine identity, the stated mechanism, and peak memory. A reviewer who has to ask for any of these will approve on vibes instead.
• Expect CI hardware to be noisier than the bench. Shared runners routinely have several times the variance of a pinned machine, so a threshold that works locally will alarm constantly in CI. Either run perf checks on dedicated hardware, or set the CI gate at a genuinely detectable effect and accept that it only catches large regressions.
• Promote each accepted patch's workload into a regression suite. Performance work that is not defended is undone within two quarters by ordinary feature development, and the agent is a good candidate for the recurring, unattended re-run — see background coding agents.
• Score the agent on expert-matching, not on speedup found. The benchmarks converged on this for a reason: an agent that reliably finds the 8% and never finds the 3× is doing a different job than the engineer you are comparing it to. Evaluating coding agents covers the harness side.