Pick an open-source eval framework by GitHub stars or by counting how many metrics it ships, and you will adopt a tool built to assert one kind of thing and then spend a quarter bending it to assert another. DeepEval, Promptfoo, Ragas and Inspect are not four attempts at the same product — they sit at four different altitudes, each fluent in a different unit of correctness. Decide what you need to call correct before you compare features, and the choice makes itself; get it backwards and the most popular tool in the category will still be the wrong one for your job.
At a glance
All four are open source and Python-first (Promptfoo runs from a Node CLI and YAML), and all four will, with enough effort, produce a number that goes red in CI. What separates them is the shape of the thing they are built to score.
| Framework | License | Core abstraction | Built to assert |
|---|---|---|---|
| DeepEval | Apache-2.0 | Pytest-style metrics on a test case | A component behaved correctly |
| Ragas | Apache-2.0 | Reference-free RAG metrics | Retrieval and grounding were good |
| Promptfoo | MIT | A declarative matrix of configs | This config beats that one |
| Inspect | MIT | Task = dataset + solver + scorer | An agent completed a task |
Read the matrix as a map of intent, not a scorecard. Every one of these is excellent at its bright column and merely adequate at the others, and the tool that is merely adequate at the job you actually have is the one that will cost you a quarter.
The altitudes, and why they do not substitute
The reason these tools do not cleanly substitute is that they answer questions at different levels of the system, and a number computed at the wrong level does not mean what you want it to. A component metric cannot tell you whether an agent completed a task; a task score cannot tell you which of two rerankers is better. The four map onto three floors — output, comparison, and trajectory — and the climb costs more compute at every step up.
DeepEval — assertions on a component
DeepEval's premise is that an LLM evaluation should feel like a unit test. You write a test case, attach one or more metrics, and it passes or fails the way pytest does — which is exactly what a developer wants in CI. Its differentiator is the breadth of its metric library: alongside RAG and conversational metrics it ships first-party agent metrics — task completion, tool correctness, argument correctness, step efficiency, plan adherence — so you can assert a specific claim about a specific step, not just a vibe about the whole run.
Where it fits
When the thing you need to protect is a component's behaviour — "the retriever still returns the right chunks," "the agent still calls the refund tool with the right arguments" — and you want that protection wired into the same CI that runs your other tests. It is the natural home for the regression-eval discipline in eval-driven development.
Where it strains
DeepEval scores outputs you hand it; it is not a harness for running a tool-using agent through a task in an isolated environment. You can evaluate an agent's trajectory by capturing it and feeding the pieces in, but if your central question is "can this agent complete the task, safely, over twenty steps with real tools," you are rebuilding what Inspect gives you.
Ragas — a specialist that stays in its lane
Ragas does one thing and refuses to pretend otherwise: it scores retrieval-augmented generation with reference-free metrics — faithfulness, answer relevancy, context precision and context recall — that decompose a RAG answer into its retriever and generator halves. Because the metrics need no gold answer, you can run them on production traffic, which is where RAG quality actually drifts.
Where it fits
When RAG is the system, or the sub-system you are hardening, and you want principled, well-understood scores rather than a general grab-bag. It is the sharpest instrument on this page for the specific job described in evaluating RAG — and it is happy to be the RAG scorer inside a larger harness rather than the whole harness.
Where it strains
Everything that is not retrieval. Ragas has grown some conversational coverage, but agent trajectories, tool-call correctness, cross-provider bake-offs and sandboxed execution are out of scope by design. Reaching for it outside RAG is choosing a scalpel for carpentry.
Promptfoo — the comparison is the unit
Promptfoo inverts the abstraction. Instead of "here is one output, score it," its native object is a matrix: prompts × models × providers × test cases, declared in YAML and rendered as a side-by-side comparison table. Its second strength is a mature red-team and security-scanning surface — adversarial test generation aimed at prompt injection and jailbreaks — which fits its "run the whole grid and see what breaks" shape.
Where it fits
Provider and model selection, prompt A/B testing, and adversarial scanning — anything where the answer is comparative and you want it as a table a whole team can read. When the question is "does the cheaper model hold up" or "which prompt variant wins," Promptfoo is built for exactly that, and it doubles as the security bake-off from prompt injection hardening.
Where it strains
Deep, custom, code-defined metrics on a single component, and long-horizon agent trajectories in a controlled environment. The matrix model is powerful for comparison and awkward when what you need is one rich assertion about one step, or a stateful agent run with real tools and side effects.
Inspect — the agent itself is under test
Inspect, from the UK AI Security Institute, is built for the heavy end. Its unit is a Task = a dataset of samples, a solver that produces an answer, and a scorer that grades it — and the solver can be a full tool-using agent, not just a single generate call. It ships sandboxed execution (Docker built-in), one interface over a long list of providers, a large library of built-in evaluations, and a log viewer built for reading trajectories. It is the framework frontier labs reach for when the agent's whole behaviour, safety included, is the thing being measured.
Where it fits
Evaluating an agent as an agent: tool use over many steps, in isolation, scored end to end, across providers, at a scale and rigour that stands up to public and safety-critical scrutiny. This is the home of trajectory and process evaluation and the multi-provider, sandboxed rigour that safety red-teaming demands.
Where it strains
Being lightweight. Inspect's power comes with more concept and setup than a developer who just wants a red/green assertion on one output in CI is looking for. For a quick component regression, it is more machine than the job needs — which is precisely why the tools compose rather than compete.
The real selector — and why you will run more than one
Stop asking which framework is best and ask what you need to assert correct on this pull request. The unit of correctness maps almost one-to-one onto the tool:
And that is the second point hiding behind the first: a serious system asserts correctness at several levels, so the mature answer is usually a stack, not a winner. A common shape is Promptfoo or DeepEval running fast component and comparison checks on every commit as a CI gate, Ragas scoring the RAG sub-system against production traffic, and Inspect carrying the heavyweight agent and safety evaluations that run less often but decide whether a new model is allowed near users. They read each other's traces more than they replace each other. The team that treats the choice as exclusive is the team that will, six months in, quietly bolt a second tool onto the first — having paid to learn the lesson this paragraph is selling for free.
When to pick which
| Your situation | Start with | Because |
|---|---|---|
| Component regression in CI, agent or RAG | DeepEval | Pytest-style assertions and first-party agent metrics |
| Hardening a RAG pipeline on live traffic | Ragas | Reference-free retrieval and grounding scores |
| Choosing a model or prompt, or red-teaming | Promptfoo | A comparison matrix and adversarial scanning |
| Scoring a tool-using agent end to end | Inspect | Sandboxed task harness, provider-agnostic, at scale |
| All of the above (most real systems) | A stack | Each asserts a different unit of correctness |
FAQ
Which of these is best for evaluating agents specifically?
It depends on which agent question you mean. For asserting a specific step — the agent called the right tool with the right arguments — DeepEval's agent metrics are the fit. For scoring a whole trajectory of a tool-using agent running in a sandbox, Inspect is built for exactly that. "Agent evaluation" is two different jobs, and they land on two different tools.
Can I just use one framework for everything?
You can, and you will feel it at the edges. Each tool is native to one unit of correctness and adequate at the others; forcing one to do all four means fighting its abstraction on three of them. Most mature stacks run a lightweight tool in CI and a heavier harness for agent and safety runs.
Is Ragas obsolete now that DeepEval also does RAG metrics?
No. DeepEval covers RAG as part of a broad library; Ragas is a RAG specialist with well-understood, reference-free metrics you can run on production traffic. If RAG is your system, the specialist is often the sharper and more defensible instrument; if RAG is one component among many, DeepEval's breadth may matter more.
Where does Promptfoo fit if I already have DeepEval in CI?
Promptfoo answers comparative questions DeepEval is awkward at — which model, which prompt, and what an adversary can break — as a matrix a whole team reads. Many teams run DeepEval for component regressions and Promptfoo for model and prompt selection plus red-teaming, with no conflict.
Do I need Inspect if I am not a safety lab?
Only if your unit of correctness is a full agent trajectory. If you build real tool-using agents and need to score whether they complete tasks over many steps in isolation, Inspect's harness earns its weight regardless of whether "safety" is in your job title. If you only score outputs, it is more machine than you need.
Further reading
On this wiki:
- Evals, in plain words — the vocabulary underneath all four tools.
- Trajectory & Process Evaluation — scoring how an agent got there, not just the answer.
- Evaluating RAG — the job Ragas is built for.
- Eval-Driven Development & CI — where the lightweight tools live.
- LLM-as-Judge for Agents — the scorer most of these lean on, and its failure modes.