Lakera Guard vs Prompt Guard 2 vs Prompt Shields vs ProtectAI DeBERTa: every ranking is self-reported

At a glance

Four detectors that get recommended interchangeably, read against what you can verify about each.

Read the last column first. One of these four publishes a benchmark, and it is the benchmark on which all four are compared. That is not an accusation of bad faith — Lakera built PINT precisely because the public injection datasets were contaminated, and open-sourcing the harness was the right instinct. It is a statement about what kind of evidence the ecosystem has: one shared ruler, held by a competitor.

The shared ruler, and who holds it

What PINT is

The Prompt Injection Test benchmark exists for a good reason. Public injection datasets leak into training sets, so a classifier fine-tuned on them scores well on them and poorly on anything else; PINT's stated purpose is to evaluate detection systems without relying on datasets those systems could have optimised against. Third-party summaries put the dataset at 4,314 inputs, English and non-English, mixing injections from public sources with Lakera's own proprietary collection, and — this part is genuinely good design — benign inputs that include hard negatives, text that looks like an injection and is not.

Three things in the methodology that change how you read the chart

The repository's methodology notes are specific, and the specifics matter more than the ranking. Azure Prompt Shields is counted as a positive if either its user-prompt shield or its document shield flags an attack, which is a reasonable choice for a product with two endpoints and a looser bar than a single-classifier verdict. Lakera Guard's own runs used only the prompt-attack category, against API version 2.0.106 dated 2 May 2025. The open-weights comparators are scored on label: ProtectAI's DeBERTa counts a positive when it emits INJECTION, Meta's Prompt Guard 1 and 2 when they emit injection or jailbreak.

Then the dates. Those methodology notes are over 500 days old. The changelog records Azure Prompt Shields being added to the results table in April 2024 and the public and internal prompt-injection categories being merged in December 2025 — a change to what the benchmark measures, after most of the figures in circulation were produced. Nothing here is hidden; it is all in the repository. It is simply not what a reader takes away from a bar chart.

The figure that does not match

A 2025 arXiv paper on evasion attacks against injection and jailbreak detectors reports Lakera Guard at 92.5461% on PINT, which it frames as 7.5439% of inputs falsely classified. The commonly quoted vendor figure is about 95.2%. Both are plausible readings of a moving target — different API version, different category selection, different snapshot of a dataset that has since been restructured — and that is the point worth taking away rather than any particular adjudication. When two careful parties report the same detector on the same named benchmark and differ by nearly three points, the benchmark is not a ruler. It is a photograph of a configuration.

The number that decides the deployment is missing

Ask any team that has turned a detector off why they did. Almost nobody says it missed an attack. They say it blocked a customer. False positives are what get a guardrail disabled, and they are also the thing the ecosystem measures worst.

There is a benchmark for it. InjecGuard (arXiv 2410.22770) introduced NotInject: 339 benign samples deliberately packed with injection trigger words like "ignore", on which the paper reports state-of-the-art detectors collapsing toward chance — open-source models including Deepset, Fmops, Prompt Guard and ProtectAI v2 all below 60% over-defense accuracy. The paper's own system reports 87.32% over-defense accuracy, 85.74% benign accuracy and 77.39% malicious accuracy, which is the shape you would expect from a model trained with over-defense in mind.

Now look at what happened to those numbers afterwards. Later write-ups in the same line of work report Llama- and Phi-based defences above 90% on NotInject, with LlamaGuard3 at 99.71% — higher than the paper that introduced the benchmark. A separate vendor post measures 45.4% over-defense on its own system, 154 of the 339 blocked, and then says the thing everyone else leaves out: NotInject is adversarially constructed to bait over-blocking, so a score on it is not a production false-positive rate.

That caveat dissolves the whole exercise, correctly. A set of 339 sentences engineered to contain trigger words is a stress test of a specific failure mechanism, not a sample of your traffic. If your agent reads support tickets, the hard negatives in your corpus are customers quoting error messages and pasting configuration. If it reads code review comments, they are developers writing "ignore the previous approach". Neither distribution is NotInject's, and neither is the other's.

// The measurement that is actually decisive, and which you own:
//   1. 2,000 real inputs from last month's traffic, unlabelled.
//   2. Run each detector. Count flags.
//   3. Read the flagged ones. Every flag on benign text is a
//      support ticket you will receive in production.
//   4. Now add the 50 injections you care about and count misses.
//
// Step 3 before step 4. The miss rate decides whether to deploy;
// the false-positive rate decides whether it stays deployed.

This is the same reasoning the wiki applies to any detector you did not train: a published accuracy number is evidence about the benchmark's distribution, and your deployment decision needs evidence about yours. See evaluating guardrails and detectors for the harness, and decision-model thresholds for what to do with the score once you have it.

Lakera Guard — the hosted API that owns the ruler

What you get

A detection endpoint with latency in the tens of milliseconds, reported in guides as a 5–50 ms range in synchronous deployment, and the most mature product packaging of the four. Lakera has been part of Check Point since the 2025 acquisition, which matters less for the classifier than for the procurement path — it arrives as a security vendor's component rather than a library.

What you are accepting

Every input the detector screens leaves your network. For an agent, that is not a subset of your data; it is your prompts, your retrieved documents and your tool results, which is close to everything sensitive the system touches. Teams under data-residency constraints usually discover this at the architecture review rather than the pricing page — the residency question is the one that decides whether a hosted detector is available to you at all, independent of how well it scores.

You are also accepting a vendor-reported accuracy claim as your primary evidence. The detection rate quoted in secondary guides, above 98%, is the vendor's; the PINT figure is on the vendor's own benchmark. That is not disqualifying — it is a normal commercial situation — but it means the reproduction step is on you, and the hosted shape makes reproduction easy, which is the fair trade here.

Llama Prompt Guard 2 — weights you can hold, with an asterisk

What you get

Two classifier sizes, 22M and 86M, released on 29 April 2025 alongside Llama Guard 4 and LlamaFirewall in Meta's PurpleLlama repository, and still the current Prompt Guard release there as of October 2026. Eighteen months without a successor is itself information: this is a stable, cheap, local classifier, not a product on a release cadence. The 22M exists because a detector that runs on every tool result has to be affordable, and a 22M-parameter model is affordable in a way an API call is not.

What you are accepting

The licence, and the fact that reasonable people disagree about it. Sources describe the weights variously as behind Meta's access request, and as open weights under a use-restricted community licence that is not OSI-approved; an October 2026 GitHub issue in a tool-result scanner project lists Prompt Guard 2 as the non-permissive option next to ungated alternatives. If your distribution model involves shipping the detector to customers or embedding it in an open-source product, that distinction is load-bearing and you should read the licence rather than a comparison table — including this one.

You are also accepting the PINT ordering's bottom half. At about 78.8% for the 86M on a benchmark it did not design, Prompt Guard 2 is not competitive with the hosted options on the only shared number available — and the honest reading of that is partly that a 86M-parameter classifier is a different class of object from a vendor's full detection pipeline, and partly that we do not know, because nobody has rerun it on current versions.

Azure Prompt Shields — two endpoints, one cloud

What you get

A managed service inside Azure AI Content Safety with separate shields for user prompts and for documents — which is, structurally, the most agent-aware design of the four, because it acknowledges that the dangerous text usually arrives in retrieved content rather than in what the user typed. If you already run on Azure OpenAI, it is a configuration change rather than an integration.

What you are accepting

A hard cloud boundary: the detector exists where your inference does, and nowhere else. And the thinnest evidence of the four — documentation describes the categories and the API, not the accuracy. Its PINT figure of roughly 89.1% comes with the methodology note that a flag from either shield counts, so the number describes the pair, not a single decision you can tune.

ProtectAI DeBERTa v2 — the permissive baseline, and its 2026 successors

What you get

deberta-v3-base-prompt-injection-v2, fine-tuned from DeBERTa-v3-base and released in April 2024, Apache-2.0 and ungated. It is the thing you reach for when the requirement is "a classifier we can ship, read, fine-tune and pin", and it is still carried as a baseline in benchmark tables published in late 2026 — including the ones where it loses, which is how you know it is being measured rather than marketed.

What you are accepting

A 2024 model against 2026 attacks, and that is the real cost rather than any particular score. It also sits below 60% over-defense accuracy on NotInject in InjecGuard's table, which — with the caveat above about what NotInject is — at least tells you the trigger-word failure mechanism is present.

The part a comparison written six months ago would miss

The permissive-licence corner of this space moved in late 2026. Horizon Labs published a Prompt Injection Guard family (a 308M base and a smaller variant) under Apache-2.0 in September–October 2026, ungated, with a model card stating it was trained only on permissively licensed data — and self-reported figures of 0.887 macro average against 0.540 for Prompt Guard 2 86M and 0.637 for ProtectAI v2. Patronus's Wolf Defender is another Apache-2.0 entrant. Treat those numbers exactly as you should treat every other number in this post: self-reported, on the entrant's own evaluation, and worth reproducing before acting on. But the structural fact is checkable and it favours them — if you need permissively licensed, ungated weights in 2026, the choice is no longer "ProtectAI v2 or accept Meta's licence".

Where the detector sits, which matters more than which one it is

Compare the four on placement and the ranking reshuffles. All four were built, and are all benchmarked, on a single input string: a user prompt, screened once, before the model sees it. That is where PINT's inputs come from and it is where the vendor documentation starts. An agent's exposure is somewhere else entirely — in the tool result, the retrieved chunk, the page the browser loaded, the issue body the coding agent read — and screening that path is a configuration decision each of these four leaves to you, with different costs. A hosted API on every tool result adds a network round trip per step and exports every retrieved document; a 22M local classifier adds single-digit milliseconds and exports nothing; Azure's document shield is the only one of the four whose API shape names the case.

The economics follow the same split. A detector on the user prompt runs once per conversation. A detector on tool results runs once per step, and an agent takes many steps, so the per-call figure that looked negligible in a chat product is multiplied by your step count — the same arithmetic as perception cost per step. This is the axis on which the open-weights options win outright, and it has nothing to do with accuracy.

Then there is what detection cannot do at all, in any position. A 2025 arXiv empirical analysis of evasion attacks against injection and jailbreak detection systems found the published systems bypassable by character- and word-level manipulations; Overflip (arXiv 2609.15013) documents repetition-induced label flips in guardrail models — the same content, repeated, changing the verdict. Both results say the same thing about architecture rather than about any product: a probabilistic classifier is a filter that raises attacker cost, and it is not a boundary. The boundary is what the agent is allowed to do after the text gets through, which is blast radius, egress control and the lethal trifecta — and a team that buys a detector instead of those has bought the wrong thing competently.

When to pick which

Further reading

On this wiki:

• Evaluating guardrails and detectors — the harness for the afternoon described above.
• Prompt injection 101 — why the direct and indirect cases need different controls.
• Prompt injection: direct and indirect — the layered defence this post's detectors are one layer of.
• Prompt-injection defence in 2026 — what the layers look like when assembled.
• The lethal trifecta — untrusted input, private data and a way out.
• Decision-model thresholds — turning a score into an action.

Sources:

• lakeraai/pint-benchmark — the benchmark, its methodology notes and its changelog.
• InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Models — the NotInject dataset.
• Bypassing LLM Guardrails: An Empirical Analysis of Evasion Attacks — the evasion result.
• Overflip: Repetition-Induced Label Flips in Guardrail Models — repetition flipping a verdict.
• meta-llama/PurpleLlama — Prompt Guard 2 and LlamaFirewall.
• protectai/deberta-v3-base-prompt-injection-v2 — the Apache-2.0 baseline.