Agentic AI Wiki
AI Blog / promptfoo vs DeepEval vs Inspect AI: Three Harnesses That Disagree About What an Eval Is
AI Blog

promptfoo vs DeepEval vs Inspect AI: Three Harnesses That Disagree About What an Eval Is

All three READMEs describe the same job — run cases through a model, score the output, fail the build. But at the level of their core data structure they disagree about what an evaluation is: an attack, an assertion, or an experiment. Pick the wrong noun and the tool will not let you write the test you actually need.

By Agentic AI Wiki 16 min read

Three open-source eval harnesses, three README pages describing the same job: run cases through a model, score the output, fail the build. Choose on that description and you will be six weeks in before you find the test you cannot express — because promptfoo, DeepEval and Inspect AI disagree, down at the level of their core data structure, about what an evaluation is. One thinks it is an attack. One thinks it is an assertion. One thinks it is an experiment. Pick the noun first; the tool follows from it.

At a glance

Three projects that all install in a minute and all claim the word "eval", built by three organisations with three different reasons to care.

ProjectStewardUnit of workLicence
promptfoo OpenAI (acquisition announced March 2026) A declared scan over a config file MIT
DeepEval Confident AI A pytest test case with metric assertions Apache 2.0
Inspect AI UK AI Security Institute A Task: dataset, solver, scorer MIT

The stewardship column is not trivia. A safety institute, a developer-tools company and a frontier lab want different things from an eval framework, and every roadmap decision in the next two years will be made by someone answering to one of those three.

Capability matrix across the three eval harnesses A three-by-five grid scoring promptfoo, DeepEval and Inspect AI on adversarial case generation, CI ergonomics, agent trajectory detail, sandboxed code execution, and reproducible run logs. Each project is strong in the columns its theory of evaluation cares about and weak in the others. Where each one leans hardest Adversarial generation CI ergonomics Trajectory detail Sandboxed execution Reproducible log promptfoo Generates the cases Exit code, config-driven Black box by design Out of scope Cases move run to run DeepEval Hand-written only Native pytest Agentic metrics Yours to build Test report, not a record Inspect AI Possible, manual Works, but heavier Whole run in the log Docker by default The whole point Strong — this is what the design is for Workable Against the grain
No row is strong everywhere. Each project's weak column is the one its theory of evaluation does not care about.

The three nouns

The unit of work in each framework Three columns comparing what each framework makes you write first: promptfoo takes a declared config and generates attacks, DeepEval takes a hand-written case and asserts a metric threshold, Inspect AI takes a dataset and runs it through a solver and scorer into a log. What each framework makes you type first promptfoo an attack You declare a target and a risk category; the cases are manufactured for you. Answers: what did I fail to think of? DeepEval an assertion You write the case, attach a metric, and the threshold either clears or fails. Answers: did this change make us worse? Inspect AI an experiment You bind a dataset to a solver and a scorer; every run emits a log you can re-derive from. Answers: can anyone else reproduce this number?
What each framework makes you type first. That object is the whole design.

Every eval framework is a bet on which question is hardest. promptfoo bets the hard question is what have I not thought to test, so its centre of gravity is generation: you declare a target and a set of risk categories, and it manufactures the cases. DeepEval bets the hard question is will this regress, so its centre of gravity is assertion: you write the case, attach a metric, and the metric either clears a threshold or fails the suite. Inspect bets the hard question is can anyone reproduce this number, so its centre of gravity is the record: a Task binds a dataset to a solver and a scorer, and every run emits a log complete enough to re-derive the result.

These are not marketing differences that a plugin can close. They determine which object you can hold in your hand. In promptfoo the first-class object is an attack strategy, so writing "the agent should refuse this and say why" is natural and writing "this metric must stay above 0.8 across the golden set forever" is a bolt-on. In DeepEval it is exactly reversed. In Inspect the first-class object is the trial, which is why it is the only one of the three that treats "run this fifty times and give me the distribution" as the default rather than a loop you write yourself.

The tool you pick decides which class of test is one line and which class is a weekend. Neither list is empty in any of the three.

promptfoo — deep dive

promptfoo architecture A YAML config declares the target providers, prompts and red-team plugins. The promptfoo generator manufactures adversarial cases from those categories, runs them against the target, and returns a local results viewer plus a CI exit code. promptfoo — you declare the surface, it writes the cases YOU DECLARE promptfooconfig.yaml Targets: prompt, endpoint, or agent behind a shim Plus the risk categories Red-team plugins Jailbreak · injection · PII Exfiltration · tool misuse Supply chain PROMPTFOO GENERATES Case generator Synthesises probes per category, then escalates across turns The test set is OUTPUT, not a fixture you keep Graders decide whether the probe landed WHAT YOU GET Findings by category Local web viewer, grouped by severity and reproducible probe CI exit code A weekly sweep against the deployed surface — not a per-PR gate Your system under test Treated as a black box with an attack surface probe / response
You declare the surface; promptfoo manufactures the cases that hit it.

What it owns

Adversarial generation. You point promptfooconfig.yaml at a target — a raw prompt, an HTTP endpoint, an agent behind a shim — enumerate the risk categories you care about, and promptfoo synthesises the probes: jailbreaks, indirect prompt injection, PII extraction, exfiltration paths, tool misuse. The cases are not a fixture you maintain. They are output.

What that buys

Coverage of the space you did not enumerate, which is the whole point of red teaming and the thing a hand-written golden set structurally cannot give you. It also means a security review that took a specialist a week can be a CI job, which is the reason this project got bought.

What it makes hard

Longitudinal quality tracking. Because the cases regenerate, run-to-run comparison is a fuzzier operation than in a fixed-dataset framework, and "did this prompt change make us 2% worse" is not the question the tool is shaped around. Teams that want both usually run promptfoo for the security sweep and something else for the quality gate — and that is a supported outcome, not a failure to choose.

DeepEval — deep dive

DeepEval architecture Pytest collects test cases written by hand. Each case attaches metrics such as G-Eval, DAG metrics or agent tool-call metrics, which call a judge model and compare the result against a threshold. The outcome is an ordinary pytest pass or fail inside the existing CI pipeline. DeepEval — the eval is an assertion, the comparator is a model YOUR TEST SUITE pytest collection Ordinary test functions, ordinary fixtures, ordinary parametrize Golden set Hand-written cases you own and version — nothing is generated METRICS ATTACH TO CASES Metric objects G-Eval — rubric judgement DAG — decision tree, so a flat score cannot hide why Agentic — tool correctness, task completion Each returns a score and compares it to a threshold WHAT YOU GET Pass or fail A red build, in the same place as every other failing test Everything CI knows Parallelism, selection, reporting, retries — inherited, not rebuilt Judge model Pin the snapshot, or your threshold moves silently grade request
An eval is a pytest assertion whose comparator happens to be a model.

What it owns

Developer ergonomics inside the test runner you already have. A DeepEval case is a pytest test; a metric is an object you attach to it; the judge call happens under the assertion. Everything your CI already knows how to do — parallelism, selection, reporting, flake retries — applies unchanged, which is a larger practical advantage than it sounds.

What that buys

The shortest path from "we have no evals" to "the build fails when quality drops". It also brings a genuinely useful metric library: G-Eval for rubric-shaped judgements, DAG metrics for decision trees where a flat score hides the reason, and agent-specific metrics for tool-call correctness and task completion. See LLM-as-judge for agents for what those judgements are worth before you gate on them.

What it makes hard

Anything that is not shaped like an assertion. Long-horizon agent runs in a sandbox, sweeps across model and parameter grids, and statistical claims over repeated trials all fit awkwardly into a framework whose unit is a single test function with a boolean outcome. You can build them; you will be building against the grain.

Inspect AI — deep dive

Inspect AI architecture An Inspect Task binds a dataset of labelled samples to a solver and a scorer. The solver ranges from a single generate call to a full tool-using agent whose model-written code executes inside a Docker sandbox. Every run writes a complete eval log holding messages, tool calls, scores and config. Inspect AI — a Task is an experiment, and the log is the deliverable DATASET Labelled samples Input, target, metadata — versioned and fixed, so two runs are comparable SOLVER Produces an answer One generate() call, or a full tool-using agent running many steps SCORER Grades the answer Text comparison up to model grading — the judge is declared, not hidden Docker sandbox Provisioned by the framework when the solver runs code Eval log Messages · tool calls · scores · model and config · sample-level detail Complete enough to re-derive the number instead of re-running on faith
Dataset, solver, scorer, log. The log is the deliverable.

What it owns

The experimental record. An Inspect Task is a dataset of labelled samples, a solver that produces an answer for each — anything from one generate() call to a full tool-using agent — and a scorer that grades it. When the solver executes model-written code, Inspect provisions a sandbox, Docker by default, so untrusted execution is a framework concern rather than yours.

What that buys

Numbers you can defend. The eval log holds the messages, the tool calls, the scores and the config, so a result can be re-derived rather than re-run on faith — which is the difference between an eval and an anecdote. The companion inspect_evals collection ships a large library of published benchmarks already implemented against the same interface, so "how do we score on the standard set" is a command rather than a project.

What it makes hard

Being casual. The Task/solver/scorer decomposition is the right abstraction for an experiment and overhead for a smoke test, and there is no five-line on-ramp equivalent to a pytest assertion. It is also the framework whose users are most likely to be evaluating dangerous-capability questions, which shows in the defaults: careful, explicit, and not optimised for the person who wants a number before standup.

Where each one hurts

Judge cost and judge drift

All three delegate the hard scoring to a model, and all three therefore inherit the same two problems: the judge is a meaningful share of your eval bill, and the judge changes underneath you when its snapshot is retired. None of the three solves this for you. What differs is whether you can see it happen — Inspect's logs make a judge swap visible after the fact, promptfoo and DeepEval will happily report a moved number without telling you the grader moved. Pin the judge model explicitly in all three, and read judge calibration before you trust a threshold.

Non-determinism

A single-run score from any of these frameworks is a sample, not a measurement, and agent tasks have enough run-to-run variance that a three-point "improvement" is routinely noise. Inspect makes repeated trials natural; the other two make you write the loop and the statistics. Whichever you pick, do not let a leaderboard-shaped HTML report talk you out of counting your runs.

Trajectory blindness

Grading the final answer of an agent that took nineteen steps tells you whether it got lucky, not whether it worked. Inspect's logs and DeepEval's agent metrics both reach into the trajectory; promptfoo's model of the target is closer to a black box with an attack surface, by design. If your failures look like "right answer, insane path", that gap is the deciding factor. See trajectory and process evaluation.

The acquisition question

OpenAI announced its acquisition of promptfoo on 9 March 2026, with the technology folding into OpenAI Frontier and the project staying open source under its existing licence. The reflexive worry — that the licence flips — is the least likely risk here and the easiest to hedge against, since MIT code already shipped cannot be un-shipped and a fork is always available.

The realistic risk is duller and harder to fork around: roadmap gravity. A red-teaming tool maintained by a frontier lab has a natural centre, and the marginal engineering hour goes where the parent's customers are. If your targets are that lab's models, this is a benefit and you should expect the integration to get better than anything an independent project could fund. If your targets are a mix including its competitors, the thing to watch is not licensing but whether non-house providers keep pace on new attack strategies. That is an observable, so observe it rather than speculating: check whether the plugins you rely on run against your providers each release.

Neither DeepEval nor Inspect is a neutral baseline either. DeepEval is the open-source surface of a commercial platform, with the usual gravitational pull toward the hosted product. Inspect is funded by a government safety institute, which is superb for durability and reproducibility and means dangerous-capability evaluation will always be better served than your customer-support bot. There is no unaligned option; there is only knowing whose problem your tool was built to solve.

When to pick which

What you are trying to dopromptfooDeepEvalInspect AI
Find the attacks you did not think ofBuilt for itHand-written onlyPossible, manual
Fail the build on a quality regressionAwkwardBuilt for itWorks, heavier
Publish a number someone will auditWeak recordWeak recordBuilt for it
Evaluate a long-horizon agent in a sandboxOut of scopePartialBuilt for it
Ship an eval this afternoonConfig, fastFastestSlowest
Compare across many models and settingsGoodManual sweepsBuilt for it

The honest recommendation for most teams is two tools, not one: a security sweep and a quality gate are different jobs with different cadences, and forcing them into a single harness is how you end up with a suite that does neither well. Run the sweep weekly against your deployed surface; run the gate on every pull request. Reach for Inspect when someone outside your team will be asked to believe the number.

FAQ

Can I run more than one of these?

Yes, and most serious teams do. The common split is promptfoo for the adversarial sweep and DeepEval for the CI gate, with Inspect added when a result needs to survive external review. They do not conflict — each reads your system through a different interface.

Does the OpenAI acquisition mean promptfoo will stop working with other providers?

Nothing announced says so, and the project remains open source under its existing licence. The thing worth monitoring is relative investment rather than removal: whether new attack strategies land for non-OpenAI providers at the same time. Check it per release against the providers you actually use.

Do I need an eval framework at all, or can I write my own?

You can write the runner in an afternoon; that is not the part you are buying. What you are buying is the metric library, the adversarial case generation, the sandboxing, and the log format — each of which is weeks of work and none of which is the product you are shipping.

How do these relate to observability platforms like LangSmith or Braintrust?

Mostly complementary. These three answer "did it pass" offline against a dataset; observability platforms answer "what happened" over live traffic and own the human-annotation and dashboard surface. Teams usually end up with one from each column.

Which one handles multi-turn agent conversations?

Inspect most naturally, since its solver can be a full tool-using agent and the whole message history lands in the log. DeepEval has multi-turn and agentic metrics that work well when your conversation is a fixture. promptfoo's multi-turn strategies exist to carry an attack across turns, not to grade a dialogue.

Is any of these enough on its own to claim compliance?

No. A framework produces evidence; a compliance claim needs a defined scope, a retained record and an accountable owner. Inspect's log format gets you closest to the evidence half, but the governance half is not a tool problem — see the governance pages below.

Further reading

On this wiki:

Project sources: