Eval-Driven Development & Regression Evals in CI

Evals as tests: the living golden set.

The load-bearing move of Eval-Driven Development (EDD) is that an eval only earns its cost when it gates. A dashboard nobody blocks a deploy on is decoration; an eval wired into CI the way unit tests gate code is a control. The mechanics are the same three lines as a test suite — treat evals as tests, treat the eval dataset as source-controlled, treat a regression as a build failure — adjusted for two things unit tests do not have to reckon with: cost (you cannot run ten thousand graded cases on every keystroke) and noise (a single run can flake for reasons unrelated to the change). The framing is corroborated across the practitioner literature, from Braintrust's release-gating posts to arXiv 2411.13768, Evaluation-Driven Development and Operations of LLM applications. The evals-101 primer is the foundation this builds on.

The dataset that does the gating is the golden set: curated, human-labeled input→expected pairs, held fixed as a stable baseline across model, prompt, and vendor changes. It is also a living artifact — the discipline is to continuously pull interesting and failing traces out of production, put them through human review, and add the validated ones back. The best source of new cases is not synthetic padding; it is your own production failures. Mine them from the signals you already collect:

• thumbs-down and explicit user corrections;
• abandoned or restarted sessions;
• escalations to a human agent;
• low-confidence or self-flagged outputs.

There is a real tension here: a dataset that grows every week cannot double as a stable baseline, because scores computed against a moving target are not comparable across time. You resolve it the way every serious test corpus is resolved — version the golden set rigorously (golden-v14, not "the golden set"), and pin the exact version each comparison runs against. A private, versioned golden set has a second advantage over a public benchmark, and it is one the benchmark-landscape essay spends its length on: it cannot leak into a training corpus and contaminate the score. Your production failures are yours.

Non-determinism and the paired significance gate.

The hard problem that separates agent CI from code CI is that agents and the models under them vary run-to-run, so a single run's pass/fail is a coin flip you have mistaken for a measurement. The first defense is samples: run k trials per case and reason over the distribution, not the point. Two summaries of that distribution matter, and they answer different questions. pass@k — the probability that at least one of k attempts succeeds — is a capability ceiling: it tells you whether the agent can do the task. pass^k ("pass hat k") — the probability that all k attempts succeed — is reliability, and it is the number production cares about. It comes from τ-bench (Yao et al.), estimated as E[ C(c,k)/C(n,k) ] over c successes in n trials. The gap between them is brutal and worth internalizing: on τ-bench an agent can average above 60% (a pass@1-flavored number) while its pass^8 falls below 25% — reliability collapses under repetition even when average competence looks fine. This is the same property HAL surfaces as its consistency dimension, and the reading-benchmarks primer is the place to internalize why pass@k and pass^k are not interchangeable.

Samples fix the measurement; they do not yet make a gate. A threshold on one candidate's noisy score will flake — some builds pass, some fail, on the same code. The gate that holds is paired and statistical: re-run both the candidate and a pinned baseline on the same cases with the same seeds, and fail the build only on a change that is statistically significant, not one that is inside the run-to-run band. The pairing is what makes it cheap and sharp — because both variants see identical cases, case-difficulty variance cancels and you are left measuring the delta the change actually caused. Say it once and put it on the wall: the CI gate is a significance test, not a threshold on one noisy number. For graded (non-binary) outputs, compare against a tolerance band rather than exact match.

eval-ci · paired regression gate — candidate vs pinned baseline
----------------------------------------------------------------
dataset: golden-v14 (218 cases)   samples/case: k=5   n=1090 runs
baseline: prod@2026-06 (pinned)   candidate: pr-3921

metric             baseline   candidate    delta
task_completion     0.842      0.851       +0.009
tool_correctness    0.910      0.888       -0.022
pass^5              0.612      0.549       -0.063
p95_latency_ms      4120       4890        +770
cost_per_task_usd   0.031      0.034       +0.003

paired bootstrap (10k resamples), same cases, same seeds:
  task_completion   +0.009   95% CI [-0.011, +0.028]   n.s.   -> ok
  pass^5            -0.063   95% CI [-0.101, -0.026]   p=0.004
----------------------------------------------------------------
GATE: FAIL — pass^5 regressed (significant), not one noisy run
      cost/latency within budget; task_completion delta is n.s.

Read the gate top to bottom. The headline task_completion went up nine thousandths — and the paired bootstrap says that delta is not significant (its confidence interval straddles zero), so it is not evidence of anything. The build fails on pass^5: a 6.3-point reliability drop whose interval sits entirely below zero (p=0.004). That is exactly the failure a single-run threshold on average score would have waved through — average competence held while consistency cracked. Cost and latency moved but stayed inside budget, so they are reported, not gated. One noisy number would have shipped this regression; the paired test caught it.

Wiring it into CI: promptfoo and DeepEval.

Run the evals on every PR and push; gate the deploy on the significance test above. Two tools have the strongest pure-CI story. promptfoo is the most CI-native of the field: declarative YAML configs, assert-style checks, and an official GitHub Action (promptfoo/promptfoo-action) that evaluates on pull requests and posts a pass/fail summary comment straight onto the PR, with red-teaming built in. The whole gate is a file you review like code.

# promptfooconfig.yaml — declarative, assert-style; runs on every PR
prompts: [file://prompts/agent.txt]
providers: [openai:gpt-5.1, anthropic:claude-opus-4-8]
tests: file://golden/*.yaml        # the golden set — source-controlled
defaultTest:
  assert:
    - type: llm-rubric
      value: resolves the ticket, invents no refund policy
    - type: latency
      threshold: 6000            # ms — a gate-able budget, not an afterthought
    - type: cost
      threshold: 0.04            # USD/task

# .github/workflows/evals.yml — the official action gates the PR
steps:
  - uses: promptfoo/promptfoo-action@v1
    with:
      config: promptfooconfig.yaml
      # evaluates the diff, posts a pass/fail summary comment

DeepEval (confident-ai/deepeval) takes the other CI-native route: it is pytest-shaped. You write assert_test() against your cases and run deepeval test run, so the evals execute in CI in exactly the invocation you use locally — no second harness to keep in sync. It ships 40-plus metrics (G-Eval, task completion, tool-use, answer relevancy, hallucination, conversational, RAG) and integrates with the Confident AI platform for storage and dashboards.

# test_agent.py — pytest-native; `deepeval test run` executes it in CI
from deepeval import assert_test
from deepeval.metrics import TaskCompletionMetric, ToolCorrectnessMetric

def test_refund_flow(case):
    metrics = [TaskCompletionMetric(threshold=0.8),
               ToolCorrectnessMetric()]
    assert_test(case, metrics)          # identical call locally and in CI

Two adjustments keep this from either bankrupting you or flaking. Budget the sample grid: run a small k on a representative subset per PR for fast feedback, and reserve the full n-samples-over-the-whole-golden-set run for a nightly or pre-release job. And keep assertions graded where the output is graded — a rubric or tolerance band, not string equality — because exact-match on a path or a paraphrase fails agents that were correct, a trap the trajectory-and-process essay unpacks in full.

Offline, online, canary, and the budgets.

Offline and online eval are not competitors; a mature 2026 team runs both. Offline is pre-deploy: batch runs against the golden set that gate releases with granular per-case metrics, so a problem is caught before any user meets it. Online is production monitoring: it adds the axes you cannot fake in a sandbox — real latency, throughput, cost, and user feedback — and it surfaces regressions live, on traffic the offline set never anticipated. The first stops bad releases; the second is the only thing that catches a release whose badness only appears against real inputs.

Guardrails are the online eval that runs in-band: pre-response scorers that block or deflect before an answer reaches the user — toxicity, bias, PII, hallucination checks that sit inside the agent loop as an interception step (see the-agent-loop for where they fit). Canary is the online eval that governs rollout: route a small percentage of traffic to the new version, run online eval on just those sessions, compare against the control cohort, and promote to 100% only if quality holds or improves — with auto-rollback the instant a regression trips. It is the significance gate from STEP 2, moved from CI to live traffic.

When you compare two versions for real — an A/B or a canary-vs-control — prefer pairwise ("arena") judging over absolute scoring. Handing the judge both outputs and asking which is better anchors it on the comparison and sidesteps the drift of an absolute grader whose internal scale wanders over weeks. The known failure mode is position bias, so guard every pairwise verdict with a position-swap consistency check; the judge-calibration essay is the full protocol for keeping the judge honest.

Drift detection and the tool landscape.

The reason online eval is non-negotiable is silent model drift. Providers update hosted models with little notice — the November-2024 snapshot is not the March-2025 snapshot — and the quality shift can land on a slice of request categories while every infrastructure metric stays flat: latency flat, error rate flat, quality quietly worse. Static pre-launch tests cannot catch a change that arrives after launch, so the only defense is post-launch monitoring plus a scheduled re-run of the golden set. The pitfall is assuming a pinned model version buys you stability; it does not, because the weights behind the name can move under you. Schedule re-baselines the way you schedule dependency updates. Dependency drift is the same threat one layer out: a changed tool, API, retriever, or upstream prompt can regress the agent without touching its own code, so re-baseline whenever any dependency changes.

The 2026 tooling has settled into recognizable lanes — pick by where your gate lives:

• promptfoo — CLI plus GitHub Action; the strongest pure-CI story, red-teaming included.
• DeepEval / Confident AI — pytest-native local-and-CI parity; 40+ metrics.
• LangSmith — experiments, datasets, evaluator templates, online eval.
• Braintrust — experiments, autoevals, online scoring, explicit release-gating framing.
• Arize Phoenix — open-source, OTel-native, self-hostable.
• Inspect / Inspect AI (UK AISI) — Task = Dataset + Solver + Scorer, sandboxed; adopted by METR and Apollo Research — the safety- and eval-lab-oriented option.
• Weights & Biases Weave — observability plus eval plus Guardrails.

The failure modes cluster, and each has a named fix. Gating on a single noisy run gives you flaky CI — use samples plus a paired significance test. An un-versioned "living" dataset gives you scores that are not comparable across time — version and pin the baseline. Judge drift corrupts online and A/B eval — go pairwise with position-swap checks. Cost and latency treated as an afterthought ship agents that pass quality and blow the budget — gate them. And a pinned model number lulls you into assuming stability that silent provider updates erode — schedule re-baselines. None of this is exotic; it is the ordinary discipline of continuous integration, applied to a system that happens to answer differently every time you ask. The teams that ship reliable agents in 2026 are the ones that stopped running evals by hand and started letting them fail the build.