The injection was ignored, not refused

An audit of about 33,000 trajectories from eleven open-weight agents found that of the indirect prompt injections that failed, 76.3% on InjecAgent and 80.3% on AgentDojo were simply ignored, while genuine refusals accounted for 18.0% and 17.2%. A low attack-success rate is therefore mostly silence, not a defended boundary — and since refusal is the only outcome that leaves a trace, refusal-rate monitoring sees roughly one in five of the attempts your agent survived. The same paper reports that changing only the system prompt moved attack success significantly for nine of eleven agents, which says the scaffold is carrying the robustness your model is being credited with.

By Agentic AI Wiki 10 min read

Your agent's prompt-injection score is mostly a measurement of silence. An audit of about 33,000 trajectories found that among the indirect injections that failed, 76.3% on InjecAgent and 80.3% on AgentDojo were simply ignored by the agent — genuine refusals were 18.0% and 17.2% — and "ignored" is the one outcome that leaves nothing behind for you to count. The number you quote as robustness is the number of attacks that did not happen to trigger, and the fix starts with measuring three outcomes instead of two.

At a glance

The paper is Ignored, Not Refused: What Low Prompt-Injection Success Rates Hide in Small Tool-Using Agents (arXiv:2610.23254, Farhan Zarif and co-authors, posted October 2026). It re-labels the failures in two standard injection benchmarks — eleven open-weight agents between 2B and 14B parameters, scored by a label-blind LLM judge — and asks what each failure actually was.

Outcome of a failed injectionInjecAgentAgentDojoHow attack-success rate records it
Ignored — never acted on, never mentioned 76.3% 80.3% A defence
Genuine refusal — recognised and declined 18.0% 17.2% A defence
Other failure modes 5.7% 2.5% A defence
What happened in the injections that did not succeed Two stacked bars, each covering the attacks that failed. On InjecAgent, 76.3 percent were ignored by the agent, 18.0 percent were genuine refusals, and 5.7 percent fell into other failure categories. On AgentDojo, 80.3 percent were ignored, 17.2 percent were genuine refusals, and 2.5 percent were other failures. Both bars are dominated by the ignored segment, which an attack-success-rate metric counts as a defence. Share of failed injections, by outcome Eleven open-weight agents, 2 to 14B parameters, about 33,000 audited trajectories InjecAgent Ignored 76.3% 18.0% AgentDojo Ignored 80.3% 17.2% Ignored Genuine refusal Other failure
Four out of five failures are the agent not reacting. One in five is the boundary doing its job.
Three outcomes and what each one tells you A three by three matrix. Ignored: counted as a defence by attack success rate, invisible to refusal monitoring, and tells you nothing about the instruction boundary. Genuine refusal: counted as a defence, visible to refusal monitoring, and tells you the boundary held. Recognised then complied: not counted as a defence, visible in the trace, and tells you the boundary is absent. Legend: informative about the boundary, not informative. What each outcome is worth Scored as a defence? Visible to monitoring? What it tells you Ignored Yes No Nothing about the boundary Genuine refusal Yes Yes The boundary held Recognised, complied No Yes, in the trace The boundary is absent Informative Not informative
Only the middle row carries information in both directions. The top row is the one the metric is mostly made of.

Why "ignored" is not a defence

The distinction is mechanical, not philosophical. An injection succeeds when the agent performs the attacker's action. It is refused when the agent processes the instruction, judges it illegitimate, and says so. It is ignored when the instruction sat in the context and never became a candidate action at all — because the tool result was truncated before it, because the scaffold's prompt kept the model narrowly on the user's task, because the model's attention never surfaced it, or because the injected instruction referenced a tool that was not in the list.

Every one of those reasons is a property of the configuration, not of the model's judgement. Change the scaffold — a longer tool result window, one more mounted tool, a system prompt that invites the model to be helpful with whatever it finds — and the same model, same weights, same attack now triggers. That is the difference between a boundary and an accident, and attack-success rate cannot tell them apart because both produce a zero.

The authors put it as directly as a paper can: small agents look secure on these benchmarks because default scaffolds leave the injected instructions untriggered, not because their instruction boundaries are sound. If you have been reading a low ASR as evidence that a small model is safe to run with tools, the evidence does not support the conclusion. It supports a weaker one: with this scaffold, on these tasks, most attacks did not fire.

The monitoring consequence, in one division

This is the part that changes something you operate tomorrow. If you monitor refusals as your signal for attempted injection — and refusal monitoring is the standard advice, because a refusal is the one event that writes itself into the transcript — then do the division on the paper's own numbers. Refusals are 18.0% of failures on InjecAgent and 17.2% on AgentDojo. So for every attempt your agent survived, you observe roughly one in five of them.

That is not a calibration error you can scale away, because the missing four-fifths are not a random sample. They are the attempts that failed for configuration reasons, which means they are precisely the population that a configuration change converts into successes. Your dashboard is blind to the attacks that are one deploy away from working.

So the instrumentation has to separate three outcomes, not two, and only one of them is free. Succeeded and refused are both visible in the trace. Ignored has to be detected: the injected instruction has to be known to you, which means a canary. Plant a benign, unmistakable instruction in the untrusted channels your agent reads — a fetched page, a ticket body, a tool result in a staging tenant — and label each run as acted-on, refused, or neither. The third bucket is your ignored rate, and it is the number that moves when someone widens a context window. Refusal monitoring in production builds the detector; this paper is the argument for why the detector needs a third bucket.

The scaffold is carrying the robustness

The paper's second result is the one with the longest reach. Holding the model fixed and changing only the system prompt moved attack success significantly for nine of the eleven agents under paired McNemar tests — the right test here, because it compares the same items under two conditions rather than two independent accuracy figures. Secondary summaries of the paper describe one configuration swing spanning 0.2% to 43% on InjecAgent; we could not confirm which agent and which pair of prompts that range refers to, so treat the specific figure as indicative and the paired-test result as the finding.

Either way the consequence is the same, and it is uncomfortable for anyone who publishes or consumes a safety number. An injection-robustness score is a property of the (model, system prompt, tool list, truncation policy) tuple, and only one of those four is what the score gets attributed to. A vendor's robustness claim measured with the vendor's scaffold tells you very little about your deployment, and your own measurement stops being valid the moment a prompt edit ships. This is the same structure as eval variance and statistical power, with prompt variance in place of sampling variance — and prompt variance is both larger and easier to introduce by accident.

Recognition is not resistance

One more finding deserves to be separated out, because it breaks a defence a lot of teams are building right now. Some agents in the audit identified the injection and executed it anyway — the paper contrasts an agent that both recognises and refuses with one that recognises the attack and still complies.

If an agent can narrate "this looks like an injected instruction" and then follow it, then a monitor that watches for recognition is watching the wrong variable. Detection and behaviour are separable, and the model's own commentary about an attack is not evidence of what it did about it. The only trustworthy signal is the action: did a tool call matching the injected intent get proposed? That is an argument for putting the check outside the model's narration, which is what monitor integrity and when the check has to act are about, and it is also why capability versus propensity is the right frame: the capability to notice tells you nothing about the propensity to decline.

What to do with this

Four things, in the order they pay off.

  • Add the third bucket. Instrument acted-on, refused, and neither, with a canary instruction in each untrusted channel. Until you have the ignored rate you do not know which of your scores is a boundary and which is a coincidence.
  • Re-run the injection suite on every scaffold change. A system-prompt edit, a tool addition, or a change to how tool results are truncated is a change to your robustness number. Gate it like a model swap, not like a copy edit.
  • Stop reading a low ASR as a defence, including your own. Report it as a pair: attack success, and the share of failures that were genuine refusals. One number without the other describes half the system.
  • Put the enforcement outside the model. The ignored-versus-refused split is the empirical case for not relying on the model's instruction boundary at all. Scope the credentials, gate the writes, and treat the boundary as a bonus — prompt-injection defence has the layers.

The honest summary of the paper is narrower than the headline and more useful: it does not show that small agents are unsafe, and it does not show that the benchmarks are wrong. It shows that the quantity those benchmarks report is a sum of two very different things, and that the larger term is the one that your configuration controls and your monitoring cannot see.

FAQ

Does this mean InjecAgent and AgentDojo are broken?

No. Both measure what they claim to measure — whether the attacker's action happened. The gap is in how the result is read: one minus attack success is reported as "defended" when it is really "did not succeed", and those are different populations with different stability.

Do the findings apply to frontier models, or only to 2B–14B open-weight agents?

The audit covers small open-weight agents only, so the specific percentages should not be transferred. The mechanism — that a failed attack can be silence rather than a decision — is not size-dependent, and the cheap move is to measure your own ignored rate rather than assume a number.

Is a high ignored rate actually bad? The attack still failed.

It failed this time, for reasons you did not design and are not tracking. A high ignored rate is a large population of attacks sitting one configuration change away from succeeding, which is a fragility rather than a defence.

How do I measure the ignored rate without running attacks against production?

Use a benign canary. The injected instruction can be something harmless and unmistakable — write a specific string to a scratch file, call a no-op tool — so you can detect compliance without creating risk, and run it in a staging tenant with the production scaffold.

If the system prompt moves the score that much, should I tune the system prompt as a defence?

Tune it, but do not bank it. A prompt that suppresses triggering is a real mitigation and a brittle one — it is undone by the next edit, and it is invisible in the version history as a security control. Write it down as one, with a test.

Further reading

On this wiki:

Sources: