Risk-Scored Permissions

8 min read

S30
Operation · Safety, Alignment & Agentic Security

Risk-scored permissions: the score is now your audit object.

A throttle that withdraws an agent's write access as a risk score climbs is better containment than any standing grant and worse evidence than the grant it replaced — because a grant is readable before the act and a score only exists while it moves. The deployable artifact is not the score; it is your band table plus a decision log carrying the score, the band and the policy version. Build those two and the control is operable; skip them and you have an authorization system whose decisions you cannot explain, reproduce or appeal.

Step 1Know which of two things you bought.

Vendors now ship two different mechanisms under one marketing phrase, and the operational work differs completely.

  • Per-task authority. No standing access; the agent receives what one task needs, for the duration of that task, capped by the authority of whoever delegated it. This is credential lifetime done right, it strictly dominates a service account, and it needs no new evidence apparatus — the grant still exists, it is just short.
  • Graduated response to a computed score. A continuously updated per-agent number drives what the agent may do right now: write withdrawn while reads continue, data narrowed, a human inserted, the runtime stopped. Microsoft shipped the first mainstream enterprise version on 19 October 2026 (Dynamic Agent Risk feeding Dynamic Conditional Access, with signals from identity, data, device, cloud, network and the agent's own actions); ServiceNow's October drift responses — redirect, restrict, deactivate — are the same shape from a different direction.

The first needs nothing from this page. Everything below is about the second, because a computed authority is a control plane you now depend on and did not previously operate.

Do not let the first sell you the second unexamined. They arrive bundled, and the per-task half is the one with no caveats. If you are time-boxed, take per-task authority now and run the score in observe-only mode until the band table and the decision log below exist.

Step 2Write the band table yourself; a continuous score is not a policy.

A score between 0 and 100 is an input. The policy is the mapping from score to capability, and if you do not author it you inherit a vendor default chosen for a median customer who is not you. Three bands is the right starting number — more than three and nobody can predict the agent's behaviour, fewer and demotion is a cliff.

Name each band by what the agent loses, not by its numeric range, because the range will be retuned and the capability contract should not move when it is:

  • Normal. Full task-scoped authority. The band the agent is in 99% of the time, and the one whose cost you are optimising.
  • Read-only. Every write, send, commit, payment and delete refused; reads and planning continue. This band is the whole point of graduated response: it is survivable for the agent and it closes the outbound leg of the lethal trifecta, which is the mechanism by which a compromised agent does actual damage.
  • Stopped. The runtime is halted and a human is paged. Reserve it for signals that indicate compromise rather than oddity, because a stop is an outage and an outage has its own incident.

Two rules that save you a bad quarter. First, pin the band boundaries explicitly and version them, so a vendor retuning its scoring distribution does not silently change your policy — the same discipline unpinned vendor defaults argues for everywhere else. Second, add hysteresis: require the score to sit above a boundary for a sustained window before demotion, and well below it before promotion. Without that, an agent near a boundary flaps between bands mid-task, and flapping is strictly worse than either band.

Step 3Log the score with the decision, or you have no evidence at all.

This is the step that is skipped, and it is the step that makes the rest defensible. Your existing audit trail records what the agent did and what was refused. It does not record the number that decided, which means the one question every review asks — why was this allowed at 14:05 and refused at 14:35? — has no answer in your own systems.

Three fields on every allowed and every refused action, not just the refusals:

# the minimum record for a scored decision
{
  "action":          "invoices.write",
  "outcome":         "refused",
  "risk_score":      78.4,        # value AT decision time, not now
  "band":            "read-only",
  "band_policy_ver": "2026-10-14.3", # which mapping was in force
  "score_inputs":    ["network", "agent_actions"]
}

Score and band are usually retrievable; the policy version is yours to emit; the contributing signals are the one you will have to fight for and the one an auditor actually wants. Set retention to cover your own review window and any statutory one, and check it against the vendor's default — a score retained for seven days cannot support a ninety-day investigation. This is a decision receipt with one extra field, and the extra field is the whole difference.

There is a governance consequence worth surfacing early: if a scored decision affects a person — a refused claim, a held payment, a blocked account — then an unreproducible score collides with contestability. "The model's risk score was elevated" is not a reason you can give someone, and it is not a reason you can defend. Either you can replay the score or that decision path should not be scored.

Step 4Test the scorer's absence before you test its accuracy.

Everyone's first question about a new detector is its false-positive rate. That is the second question. The first is what your system does when the scorer is not answering, because a static policy has a failure direction you chose and a scored throttle has one that falls out of implementation details. Run all four cases in staging before production, and write the observed behaviour into the runbook:

  • Stale. The scorer returns last known values. Agents keep whatever authority their last good score bought — including the one whose score was about to rise.
  • Missing. The scorer returns nothing. If absence maps to low risk, every agent is promoted simultaneously and silently; this is the one that turns a scorer outage into a security incident. Decide it as a fail-closed or fail-open question and make it explicit in policy.
  • Lagging. The score is correct but arrives minutes late. Measure the lag; it is the real upper bound on what the throttle can contain, and it is not in any datasheet.
  • Flapping. The score oscillates across a boundary. Hysteresis from STEP 2 is the fix; verify it by injecting an oscillating score rather than by trusting the setting.

Then test accuracy, with the methodology from evaluating guardrails and detectors — and be clear-eyed that an agent is anomalous by construction, which is why generic baselines underperform and why the trajectory-shaped signals in detecting agent compromise are the ones worth weighting.

Step 5Price the demotion: an agent that loses write mid-task is a half-finished job.

The design discussion covers whether to demote. The incident is always about what demotion did to work in flight, and this is the cost nobody budgets. An agent twelve steps into a seventeen-step job that loses write authority at step thirteen has already made twelve changes to the world and cannot make the thirteenth. If step thirteen was the one that made the previous twelve coherent — the commit, the confirmation email, the ledger entry — you have manufactured an inconsistency with a security control.

  • Make the demotion boundary a checkpoint boundary. If the loop has durable state, demote between steps and persist the position so the task can resume from where authority stopped rather than restart.
  • Require idempotent writes before you enable a throttle at all. Retry-after-demotion is a retry, and idempotency is what keeps it from double-charging somebody.
  • Route the stranded task to a queue, not to nothing. A demoted agent should leave a reviewable item behind; otherwise the containment is invisible and the work quietly does not happen, which is the failure mode users report as "it sometimes doesn't finish".
  • Keep the stop path out of band. If the band logic and the agent run in the same control plane, a compromised control plane removes both — see kill switches.

Instrument demotions as a first-class rate, alongside refusals and errors. A throttle nobody has seen fire in a month is either well-tuned or disconnected, and the two look identical from the dashboard.

Step 6The honest limits, so you buy it for what it does.

A scored throttle reduces what a compromised or drifting agent can do next. That is valuable and it is narrow. Four things it does not do:

  • It does not stop prompt injection. The injection still lands; the throttle shortens what follows. Structural defence remains breaking a leg of the trifecta, and egress allowlisting is cheaper and more deterministic than any score.
  • It does not cover agents nobody enrolled. Scoring applies to identities the platform knows. The script with an API key and a loop, the CI job, the workstation tool — unscored, and not the benign tail. Pair the throttle with directory accounts for agents and treat the enrolment gap as the finding.
  • It does not replace the access review. It removes the standing grants a review used to read, which makes reviewing agent credentials harder, not unnecessary: what you review becomes the band table and the reachable set per band.
  • It does not transfer across vendors. A score computed from one estate's telemetry is not comparable to another's and cannot be reconstructed outside it. If you run agents across clouds, treat each score as partial by construction and size the throttle to what that estate can actually see.

Minimum viable adoption, in order: take the per-task authority immediately; run the score in observe-only for two weeks and read its distribution before choosing any boundary; write a three-band table with explicit, versioned boundaries and hysteresis; emit score, band and policy version on every decision and set retention to your review window; then inject a dead scorer and an oscillating scorer in staging and record what happened. Only after that should the throttle be allowed to refuse a real write. The progressive autonomy ladder is the human-facing analogue, and its rule applies here too: demotion must be automatic and reversible, and the current rung must be impossible to misread.