Detecting Agent Compromise

Content inspection is prevention, and prevention has no detection story.

The standard stack is an input classifier scoring retrieved text for injection, plus an output filter looking for exfiltration patterns. Both are worth having. Neither tells you that an agent has been compromised, and the reason is structural rather than a question of model quality.

• The classifier's miss is silent. A detector at 98% recall is a good detector and it still passes one attempt in fifty with no event emitted. There is no second signal, because the same component decides both "is this an attack" and "was there an attack".
• After the fact, the instruction is indistinguishable from context. Once injected text is in the transcript it is just tokens the model read, and every subsequent step looks like the model reasoning over its context — which is exactly what it is. Scanning later steps for the injection finds nothing, because the injection already worked and has no reason to persist.
• Output filters are downstream of the interesting exfiltration paths. A markdown image URL that the client fetches never passes through your text filter as text; see rendering agent output safely.

Your agent is anomalous by construction, so generic baselines are worthless.

Point an off-the-shelf user-behaviour analytics product at a service account that happens to be an agent and it lights up permanently. The agent reads hundreds of documents an hour, calls a dozen APIs in sequence, works at 3 a.m., and its volume varies by an order of magnitude with the task it was given. Every one of those is a textbook insider-threat indicator, and every one of them is correct behaviour. Teams respond by widening the thresholds until the alerts stop, at which point the control is off.

The fix is to change the unit of baselining. The comparison that discriminates is not this identity versus its history — it is this run versus other runs of the same task type. A run of summarise-ticket has a characteristic shape: roughly this many steps, these tools in roughly this order, this breadth of reads, egress to these hosts. That distribution is tight, because the task is fixed, and a run that leaves it is interesting even when the identity's rolling averages look normal.

• Tag every run with its task type at dispatch. If you cannot, you have no denominator and nothing below works.
• Build the baseline from successful production runs, refreshed on a rolling window, not from a design document about what the agent should do.
• Keep per-task-type baselines separate per tenant where tenant data shapes differ enough to matter; otherwise your largest customer defines "normal".

Five signals that discriminate, in order of how much they earn.

These are ranked by signal-to-noise in practice, not by how sophisticated they sound. The first one is nearly free and catches the majority of successful injections, because a hijacked agent has to spend its remaining steps on the attacker's goal rather than the user's.

• Task abandonment. Compare the run's terminal state against the task it was dispatched with: did it produce the artefact it was asked for? An injected agent typically returns a plausible-looking non-answer, or succeeds at something adjacent. Cheap to compute, and unusually hard for an attacker to avoid — completing the user's task as well costs the attacker steps and context.
• Tool-sequence novelty. Score the run's sequence of tool names against the n-grams observed for that task type. A read → read → summarise task that suddenly contains read → list_keys → http_post scores as unseen. Bigrams over tool names catch most of it; you do not need a sequence model.
• Egress destination novelty. First contact with a host, first use of a tool that can reach outside, an unusual ratio of bytes out to bytes in. This is the highest-severity signal and the lowest-volume, and it belongs next to egress control — where the allowlist blocks, the detector records what tried.
• Read-set expansion. The count and diversity of documents, rows or repositories touched relative to the task type's distribution. Staged data collection before an exfiltration attempt shows up here first, and it is also how a retrieval permission bug announces itself.
• Authorisation friction. Denied calls, scope errors, refused tools, retries against the same forbidden endpoint. A healthy agent rarely gets told no twice for the same thing; something probing its boundaries does. Log denials as events in their own right — most stacks throw them away at the policy layer.

None of it is computable unless the loop emits the right fields.

Most agent traces are built for debugging: prompts, completions, latencies. Detection needs a different projection of the same run — a compact per-step record whose fields are comparable across runs. Emit it from inside the tool-call path, not from a wrapper the agent's own errors can skip.

# detect/step_record.py — emitted on every tool call, before dispatch
def record(run, step, tool, args, result):
    emit({
        "run_id":      run.id,
        "task_type":   run.task_type,      # the baseline key
        "principal":   run.on_behalf_of,   # human, not service account
        "step":        step,
        "tool":        tool,               # name only — the n-gram unit
        "dest":        egress_host(args),  # None for local tools
        "read_ids":    hashed(result.ids), # breadth, never content
        "bytes_out":   size(args),
        "denied":      result.denied,      # keep policy refusals
        "ctx_source":  args.provenance,    # which retrieved doc drove this
    })

Two fields carry more than their weight. principal keeps the human the run acts for, so a compromise can be scoped to the people affected rather than to the one service account everything shares — the argument in agent identity. ctx_source keeps the provenance of the text that preceded the step, which is what converts "this run went sideways" into "this document turns agents", and that is the finding that protects every other tenant. Hash identifiers rather than storing content; the detector needs breadth and shape, and retaining the documents themselves turns your detection store into the richest target on the network. Retention follows trace sampling and retention — and note that sampling a percentage of runs is exactly wrong here, because the run you dropped is the one you needed. Sample spans for cost, keep the step record for every run.

A detector that only opens tickets has not shortened anything.

Detection buys you nothing until it is wired to an action that outruns a human. Agents complete their remaining steps in seconds; a page that arrives in fifteen minutes arrives after the run finished. Build a ladder and put the cheap rungs in the loop itself.

• Degrade. On a moderate score, drop the run to read-only tools and finish. Most false positives end here costing one degraded run, which is what makes an aggressive threshold affordable.
• Gate. On egress or authorisation signals, require approval for the next state-changing call. The approval surface already exists if you have human-in-the-loop; this reroutes into it rather than building something new.
• Halt. On a high score, stop the run and checkpoint it. This is the kill switch at run scope, and it needs to be callable by the detector without a human in the path.
• Quarantine the source. When ctx_source points at a specific document, block it for every run pending review. One poisoned page reaches every agent that retrieves it, so the containment unit is the document, not the run.

Set thresholds by what a false positive costs at each rung, not by a fixed percentile. Degrade can afford to fire on a few percent of runs; halt cannot. Route confirmed hits into incident response, and expect the first cleanup to be partial — a channel closed at the tool layer is often reopened somewhere less obvious.

Measure mean time to detect, or you are guessing.

An unexercised detector is a belief. The measurement is cheap: plant benign canaries and count steps, not minutes.

• Put a canary instruction in a document the agent will retrieve — one that asks it to call a harmless internal endpoint you control. Run it weekly against production configuration in a staging tenant. Your metric is the number of steps between the agent reading it and the detector firing.
• Track detection stage. Which rung caught it, or whether the run completed uncaught. Detection-at-halt and detection-after-completion are different products.
• Vary the payload. One that exfiltrates, one that quietly abandons the task, one that escalates scope. Each exercises a different signal, and the abandonment case is the one most detectors fail.
• Report both numbers. Rate of compromise from the classifier, duration from the behavioural stack. A programme reporting only the first has no evidence about what happens after a miss.

Related: prompt-injection defence for the prevention half, data exfiltration risks for what the egress signal is protecting, failure taxonomy and triage for separating compromise from ordinary failure, and decision receipts for the record you will want when explaining what happened.