Human-in-the-Loop & Least Privilege

10 min read

S5
Operation · Safety, Alignment & Agentic Security

Human-in-the-loop & least privilege: bounded autonomy by design.

Most teams place their approval gates roughly correctly and still lose them, because a gate is only worth its latency if three separate things hold: it shows the effect rather than the model's description of its intent, it cannot be satisfied through the channel that carried the attack, and someone has the time to decide it. Nine public advisories against one coding agent's approval path between March and October 2026 contain no mistakes about which actions to gate. Least privilege remains the foundation for a reason that is easy to miss — it is the only control that reduces how many gates you need.

Step 1Least privilege first, because it sets the gate count.

Least privilege is the oldest principle in security and the highest-leverage one for agents, because it attacks impact rather than chasing triggers. Every prompt-injection essay ends in the same place: the attack that matters is the one that produces a harmful action, and an agent that lacks the capability cannot perform the harmful action no matter how thoroughly its context is poisoned.

Applied to agents, it has four dimensions, and each one removes work from the human rather than adding it:

  • Tool minimality: the agent has only the tools this task needs, not the full catalog "in case". A tool absent from the schema needs no gate.
  • Scope minimality: each tool's reach is narrowed — read-only where possible, row- or record-limited, destination-allowlisted, single-tenant. A refund tool capped at $50 turns a gated action into an ungated one.
  • Credential minimality: short-lived, per-task, narrowly-scoped credentials; no ambient long-lived admin token in the agent's environment. See scoped credentials and credential lifetime.
  • Temporal minimality: a capability is present only during the state where it is valid, then withdrawn — which is also what stops a grant made for one task from financing the next.

The reason to do this work before touching the gates is arithmetic, not purity. Gates consume a fixed budget of human attention, so the number of gated actions per hour is a resource you spend; narrowing scope is how you stop spending it. Every action that becomes safe to run unattended is an action that never competes for a reviewer's ninety seconds — the budget itself is oversight capacity, and blast radius is the cheaper lever on it.

The fastest security win for most production agents is still closing the gap between "tools the agent has" and "tools this task actually requires". It shrinks every impact category at once, needs no model changes, and it is the only intervention here that makes the human's job smaller instead of larger.

Step 2Place the remaining gates by consequence.

A human reviewing each step is a slow but extremely strong control: a competent reviewer rejects the email to attacker@example.com that an injected agent was about to send. Multi-step autonomy is valuable precisely because it removes that reviewer — and removes the control with it. The job is to re-introduce the checkpoint selectively, by consequence rather than by step count: require approval when an action is irreversible, externally visible, privilege-changing, or high-value, and let reversible, internal, low-value actions run.

# Consequence-based gating, not step-based
read_internal_doc        # auto — reversible, internal
draft_reply              # auto — no external effect yet
send_external_email      # GATE — externally visible
delete_records           # GATE — irreversible
grant_access / merge_pr  # GATE — privilege / production
issue_refund > threshold # GATE — high-value

This rubric is not where deployments fail. It is published in roughly this form everywhere, it is easy to apply, and in the field it is usually applied about right. What follows are the three ways the gate you placed correctly stops being a control, in rough order of how often it happens.

Step 3Failure 1: the approval is bound to an intent, not an effect.

In most agent products the string the human approves was written by the model, describing what it means to do. The code that will act never saw that string. So the approval is a decision about a claim, and the gap between the claim and the effect is exactly where the surprises live — not because the model lies, but because it does not know what the call will touch.

A worked example. A support agent proposes "archive the duplicate thread". The reviewer approves a sentence; the API call archives a thread, cascades a label to its two children, and fires a webhook to the customer's CRM. Nothing in the proposal was false and nothing in it was reviewable:

# What the human approved (model-authored):
"Archive the duplicate thread so the queue stays clean."

# What the code computed, which nobody saw:
archive(thread=8812)        1 thread
  ├─ cascade label          2 child threads re-tagged
  └─ webhook  crm.acme.com  outbound, externally visible

The fix is the distinction kubectl has drawn for years between --dry-run=client and --dry-run=server: a preview is worth an approval only when the same code path produces the preview and the action, and when the approval token is bound to that computed effect rather than to the request behind it. Get that binding right and the decision also becomes much cheaper — the reviewer reads a diff instead of reconstructing one, which is the single largest lever on how many gates you can afford. The detail is in dry runs and preview modes, and the interface half is approval and confirmation UX.

An approval gate the user cannot meaningfully evaluate is worse than none: it manufactures assurance and shifts accountability to a human who never had the information to decide. If you cannot render the computed effect, you have not built a gate — you have built a consent dialog, and the thing it collects consent for is unspecified.

Step 4Failure 2: the gate is reachable from the channel that carried the attack.

A gate is a reference monitor, and a reference monitor that the attacker's input can influence is a formality. Three versions of this show up in production, and the first is the most common security bug in the whole pattern.

The pre-approval list that parses differently from the thing that executes. Nine GitHub advisories were published against Roo Code's approval path between 30 March and 13 October 2026 — five of them on a single October day — with researchers naming seven different shell constructs: redirects, glob tokens, quote boundaries, comment boundaries, the pipe-both operator, command substitution, and output used as a command. Every individual fix was correct. The class is not fixable by better matching, because the gate was parsing a language owned by bash: CWE-436, interpretation conflict. Approve the object you will execute — usually an argv vector — and the class disappears. The full argument is in auto-approval and command allowlists, with the field evidence in the allowlist and the shell disagreed.

The approval the agent can synthesise. If "yes" is a click in a surface the agent itself drives, or a reply in a thread the agent can post to, then the same injected context that chose the action can also produce its approval. High-impact actions need confirmation through a channel the in-context attacker cannot forge. This is also why the published CVSS vectors for that October cluster record passive user interaction: the developer asked the agent to read an issue, and the issue contained the command — ordinary indirect injection, and an agent holding untrusted input, private data and a shell is the lethal trifecta with the gate as its only mitigation.

The agent asked to gate itself. A model's self-assessment shares the failure mode it is meant to catch: the poisoned context that produced the bad action can produce a confident "this is safe, no approval needed". It can advise; it cannot decide. The gate decision has to be deterministic and external — a code-level policy keyed to the tool and its arguments, which is the job of policy as code, and where risk differs by argument, of risk-scored permissions.

Step 5Failure 3: nobody has the time to decide it.

The third failure is not a bug and leaves no advisory. Approval queues have no backpressure, so when escalations outrun the people who answer them, nothing stalls and nothing alarms: the reviewer silently reprices each decision from ninety seconds to twenty-five, and approval latency improves on the dashboard while the verdict stops being a verdict.

Emre Turan's Oversight Has a Capacity: Calibrating Agent Guards to a Subjective, Fatiguing Human (arXiv 2606.08919, submitted 8 June 2026) puts a mechanism under this. Model the reviewer as fatiguing and realized safety becomes an inverted U in the escalation rate — it rises, peaks, falls — and a flooding attack follows directly: bury one harmful action in a long stream of routine approvals and it is rubber-stamped, where a reviewer asked five times reads each request. The paper is honest that the inverted U and the flooding result are consequences of a fatigue model offered to motivate a human study, on 125 hand-labelled coding-agent actions from one domain. The measurement that needs no such assumption is the one worth carrying: on those actions, reviewer agreement about which were risky came out at a Fleiss' kappa of 0.52. Moderate. Three competent people shown the same agent action disagree about whether it should have stopped.

Two operational consequences. First, the escalation rate is attacker-controllable, which makes this a security property and not an ergonomics complaint — denial of attention, next to denial of wallet. Second, there is a cap, and you should decide now what happens at it. When escalations exceed what your reviewers can decide, the fleet pauses, degrades to read-only, or sheds low-risk tasks — fail closed. Auto-approving the overflow does not degrade the control; it removes it, at exactly the moment an attacker would choose.

┌──────────────────────────────────────────────────────────┐ │ A GATE THAT IS ACTUALLY A CONTROL │ │ │ │ bound to effect → same code path previews and acts │ │ unreachable → approval channel ≠ attack channel │ │ decidable → escalations/hour ≤ capacity │ │ │ │ any one missing → a log line saying a human was here │ └──────────────────────────────────────────────────────────┘

Step 6The review that tells you which one you have.

All three failures are visible in data you already keep, and the queries are small. Run them quarterly and after any change to the tool catalog.

  • Pick ten approvals from last week and reconstruct the effect. If you cannot tell from the log what the approved call actually touched, you have Failure 1 — and so does your decision receipt.
  • Ask who can cause an approval. List the surfaces that can deliver a "yes" and check each against the surfaces untrusted text reaches. Any overlap is Failure 2.
  • Plot decision time against queue depth. A spike of sub-five-second decisions, or approval probability rising with depth, is Failure 3 located empirically in your own traffic.
  • Double-review a sample. Route a handful of escalations to two reviewers independently. Moderate agreement means the dialog, not the threshold, is what needs fixing.
  • Log the pair. Proposed action, computed effect, verdict, outcome, in one row. Without the pairing a bypass is indistinguishable from a normal approval, which is how one goes unnoticed for seven months.
Question
Approval gates kill the productivity gains. How do I justify them?

Gate by consequence rather than by step and the cost collapses: the overwhelming majority of an agent's actions are reversible internal reads that run untouched, and friction lands on the handful of irreversible or externally-visible actions where an unattended mistake or a successful injection is most expensive. Then attack the cost of the remaining decisions rather than their number: that is where the leverage is, because capacity is attention divided by cost-per-decision, so halving the cost does the same work as halving the gate count and keeps the coverage. Binding the approval to a computed diff is the cheapest way to halve it — measure your own before and after rather than trusting a ratio. If the gates still dominate your latency, the honest read is that the agent's scope is too wide, not that the oversight is too heavy.

Question
Can a second model review the first one's actions instead of a human?

As a filter, usefully; as the gate, no. A reviewing model is a cheap way to raise the quality of what reaches a human — summarise, cluster, flag — and that directly raises capacity. What it cannot do is hold the decision, because it shares both failure modes that matter: the same injected content can reach it, and its verdict is probabilistic where the control needs to be deterministic. Use it to spend the human budget better, keep irreversible effects behind an external policy, and read when the check has to act before you let the reviewer take an action of its own.

Do three things this quarter. Bind your highest-consequence gate to a computed effect produced by the code path that will act, so the reviewer reads a diff. List every surface that can deliver an approval and remove any that untrusted text can reach. Then compute your escalation rate against the reviewer-attention you actually believe exists, and write down what the fleet does when the first exceeds the second. Least privilege is what keeps all three affordable: every tool you narrow is a gate you never have to staff.