Operations / Safety & Security

Safety & Security

Prompt injection, sandboxing, exfiltration, red-teaming, deployment safety — the threat model an agent's environment creates.

  1. The Agentic Threat Model
    Why autonomy and tool use widen the attack surface, and the four channels attacker-influenced text reaches an agent.
  2. Prompt Injection: Direct & Indirect
    How prompt injection works, why no clean fix exists, and the layered defense pattern for defenders.
  3. Data Exfiltration & Tool Misuse
    The confused-deputy pattern in agents: exfiltration sources, hidden sinks, and how to cut the chain.
  4. Guardrails: Filtering, Sandboxing & Scoping
    Probabilistic vs deterministic guardrails and how to layer input, output, sandbox and capability controls.
  5. Agent identity
    Who is acting when an agent calls a tool? Service accounts, on-behalf-of patterns, and the audit consequences of getting the answer wrong.
  6. Scoped credentials for agents
    Why agents should never hold human-grade credentials — short-lived, narrowly-scoped, per-action tokens, and the failure modes when you try to take shortcuts.
  7. Human-in-the-Loop & Least Privilege
    Gate placement is the part teams get right; the nine advisories filed against one coding agent's approval path between March and October 2026 contain no mistake about which actions to gate. A gate is a control only if three things hold — it shows the computed effect rather than the model's description of its intent, it cannot be satisfied through the channel that carried the attack, and escalations stay under what a reviewer can decide (reviewer agreement on which agent actions are risky measures at a Fleiss' kappa of 0.52). Least privilege stays the foundation because it is the only control that reduces how many gates you need.
  8. Red-Teaming & Safety Evaluation
    Adversarial testing of agents as a repeatable, outcome-graded pipeline gate, not a one-off session.
  9. Alignment Basics: Intent & Oversight
    Instruction-following vs intent, reward hacking, and scalable oversight as the practical builder lever.
  10. The Pre-Ship Safety Review
    A practical, fail-closed-first deployment checklist including MCP/third-party supply-chain trust.
  11. RAG Pipeline Security
    Why retrieved context is untrusted input that skipped the guard — corpus poisoning, indirect injection, embedding leakage, and the trust-boundary design that contains them.
  12. Egress Control for Agents
    Three labs disclosed models reaching real systems from an eval sandbox in three weeks, and the boundary turned out to be a sentence in the prompt — compute isolation says nothing about routing, a domain allowlist is a scope control rather than a confidentiality one, and the only workable boundary is a proxy scoped to the task.
  13. Rendering Agent Output Safely
    EchoLeak exfiltrated data from Copilot with no click and no tool call — the model was steered into writing a markdown image and the client fetched it, which means model output is untrusted input to every renderer and the leak happens downstream of every egress control you built.
  14. Vulnerability Management for Agent Platforms
    The highest-severity bugs in an agent stack arrive with no patch to apply — the vendor fixes them service-side and tells you afterwards — so the asset inventory learns nothing, the scanner cannot confirm remediation, and the only lever that changed your exposure was a credential grant made months before disclosure.
  15. Secrets Management for Agents
    An agent reads attacker-influenced text on every step, so any secret reachable from inside the loop is one crafted instruction away from a log, a tool argument, or an exfiltration URL — the fix is to move the credential out of the model’s reach entirely and have a broker attach it at the egress boundary, so the model handles the name of a capability and never the secret behind it.
  16. Denial of Wallet & Cost Attacks
    A request-per-second limit stopped bounding your spend the moment one request could fan out into an unbounded loop, so the attacker optimises amplification rather than volume and leaves every latency graph green — attach a money-denominated ceiling to each run, enforce it in the component that issues the calls rather than in a prompt, and watch cost per principal instead of aggregate spend.
  17. Permission-Aware Retrieval
    Indexing strips the container that carried the access rules, so retrieval becomes the widest read your system performs — and filtering the answer is not a control, because anything the model saw is disclosed and even the count of hidden results leaks. Enforce before the search, store permission keys rather than resolved member lists so revocation lands on the next query, late-bind the few chunks that reach the prompt, and test it with two users and a canary string.
  18. Detecting Agent Compromise
    Content controls reduce how often you are compromised and tell you nothing about when it happened, because the classifier that misses is also the thing that would have reported it — what survives is behavioural: a hijacked agent abandons the dispatched task and its tool-call sequence leaves the shape that task type produces, which only discriminates if you baseline per task type rather than per identity, since a healthy agent is wildly anomalous by ordinary SOC standards.
  19. Attacker-Operated Agents
    The 2026 intrusions that involved agents ran a commercial coding assistant, driven by a human, under credentials already stolen — no new access, no new technique, and the refusals that fired were defeated by reframing the work as an authorised penetration test. What changed is tempo and breadth under one identity, so the controls that move are credential blast radius, a tier-zero hypervisor plane and time-to-revoke, not a blocklist of AI tools you could never enforce.
  20. Insider Misuse of Sanctioned Agents
    The reports all measure shadow AI — data leaving through an unsanctioned chatbot — while the harder case runs inward through the agent you approved: an employee with ordinary permissions asks one question and gets four thousand individually-permitted reads synthesised into the document that used to take three weeks. Permission has nothing to say about it, so detect on records returned and distinct subjects touched, bind retrieval to a case id, and log the verbatim prompt or your audit trail becomes a deniability machine.
  21. Telemetry & Logs as Untrusted Input
    A WAF records the payload it blocked verbatim, so your block log is the one corpus an anonymous stranger can write to at will — and it arrives at the triage agent wearing a "security" label, which is how a disclosed technique reached a 90% success rate against a coding agent on a vendor default. Split the reader from the actor, deny egress by default, and prove it with a canary you plant in your own logs.
  22. Physical Actuation: Safety Without Undo
    Every control in the agent-safety stack assumes the action can be taken back, and a syringe, a stage motor or a robot arm gives you none of retry, rollback or sandbox — so enforcement moves below the model into a driver that cannot be argued with, and the human decision moves from approving a step to authorising a bounded envelope for a whole run. Keep a driver limit and a rated protective function clearly apart, watchdog every device into its own defined safe state, keep the emergency stop out of the software path, and measure unplanned stops and envelope exits rather than protocols completed.
  23. System-Prompt Extraction
    OWASP lists prompt leakage as LLM07 and says in the same breath that the prompt is neither a secret nor a security control — so hardening the refusal buys delay against one adversary while degrading the product for everyone, and the ruleset stays recoverable by probing anyway. Separate the text from the secrets it embeds and the tool surface it maps, give every “never” an enforcement twin outside the model, canary each version so a leaked copy names its build and tenant, and resist rewriting the prompt after a leak.
  24. Lifecycle Hooks & Harness Config
    Every control around your coding agent sits between the model proposing and a human approving, and a lifecycle hook takes neither path: a shell command bound to an event, running with the developer's full privileges, firing at moments the model never observes, shipped as settings rather than code. September 2026's HookPry results compromised all seven harnesses tested via the update path, so the exposure is version-to-version review, not installation. Enumerate the effective hook set on every host, move the config into the pipeline you trust for code, and give hooks an explicit environment allow-list and default-deny egress.
  25. Honeytokens for Agent Systems
    Behavioural detection drowns in base rates because an agent is anomalous by design; a planted token with no legitimate user restores precision — six token classes, the use-not-read rule, and a decoy that measures injection susceptibility in production.
  26. Agent Artifacts on the Endpoint
    Every agent security control you own assumes the agent is what is under attack — and meanwhile its tokens, its list of connected systems and a searchable record of everything it was ever asked sit in predictable paths on laptops you do not monitor. Gen Threat Labs published stealer collection rules on 8 September 2026 naming Claude, Cursor, Cline, Continue, Codex and OpenCode artifacts; adding the next agent is a configuration push, not an exploit. Mode 0600 is a user boundary and the malware runs as the user, so enumerate the five artifact classes, price them by what they buy, and make the file worth less rather than unreadable.
  27. Agents on the IT/OT Boundary
    Booz Allen gave two frontier models nothing but subnet boundaries and they cleared all eight attack objectives against its manufacturing testbed, reaching action inside the control environment in just over sixteen minutes; the only safeguards reported as holding were a controller that refused a command and a motor whose local-mode switch was set by hand. If an agent can reach a control’s state, that control is friction rather than prevention — so enumerate the crossing paths, rank every control by whether the network can change it, give the agent a read path and make the write path a work order.
  28. Agents with Directory Accounts
    Giving an agent its own workplace account fixes the shared robot user nobody can audit and in the same motion ships the first agent identity a stranger can write to without a credential. Treat the bundle as six separate grants, not one: attestation answers which agent acted and never whether it should have, a mailbox is an unauthenticated write channel whose SPF and DKIM authenticate a domain rather than a request, and group membership is the real blast radius — add the agent to engineering@ and it transitively inherits six months of sharing nobody decided to give it. There is also no joiner-mover-leaver event for something that was never hired, so the orphaned agent account is the default end state.
  29. Third-Party Agents Logging In As Your Users
    Five large consumer businesses made blocking third-party agents a stated policy inside four weeks of October 2026, and not one can reliably detect what it says it blocks: every signal is either voluntary (a User-Agent, a bot-auth header) or an inference a determined builder smooths out next sprint — and a detector tight enough to catch a careful agent starts blocking screen-reader users. So split the three things you might be refusing, because only two are enforceable: the channel is not, the capability is perfectly (an operation absent from a surface cannot be invoked through it), and the counterparty is because you issued the grant. Then publish the sanctioned path, or your block is a feature request for a workaround — Schwab kept reads open through its own MCP servers while barring money movement, and Yelp admits agents whose makers hold a contract.
  30. Risk-Scored Permissions
    A throttle that withdraws write access as a risk score climbs is better containment than a standing grant and worse evidence, because a grant is readable before the act and a score only exists while it moves. The deployable artifact is your own three-band table — named by what the agent loses, with versioned boundaries and hysteresis — plus score, band and policy version on every decision, allowed ones included. Test the scorer's absence before its accuracy: stale keeps authority nobody re-earned, and missing treated as low risk promotes every agent at once. Then price the demotion, because an agent that loses write at step thirteen of seventeen has made twelve changes it cannot finish.
  31. When the Credential Broker Is Compromised
    Moving provider keys out of the model's reach and into a broker at the egress boundary is the right fix, and it builds one process holding the union of every credential you own — which stopped being hypothetical on 21 October 2026, when a compromised LiteLLM release put the upstream provider credentials in the proxy's own database in scope. Write the sentence first (N credentials, M providers, T tenants, widest does X), hold the narrowest credential each provider will issue because expiry does real work against an attacker who exfiltrates the store to use later, treat the broker's release channel as part of the security boundary (a provenance attestation verifies the pipeline, not the source), and measure rotation with a stopwatch. The compensation: the broker sees every call before it leaves your network, so a destination allowlist on that one host is the cheapest high-signal detection in the stack.
  32. Authorised Agentic Testing
    When a third party's security agent goes somewhere it should not, the only evidence it was ever authorised is the operator's word: in the WorkOS disclosure of 30 October 2026 the timestamps matched and an independent forensics firm agreed, and the claims that the agent had escaped its test environment and that the copied data was deleted still rested on one unnamed person. A rules-of-engagement template written for people has no field for identity, rate or the tester's own containment, and no regulator has published one. So mint the credential yourself, write the clause that unmarked traffic from the engagement is treated as an intrusion without consultation, cap reads and records separately at your edge so the limit also pages someone, and negotiate the named deletion signatory before the test rather than after the incident — a user-agent string is a self-declaration, not a control.
  33. Local MCP Servers on the Endpoint
    Neither transport MCP offers gives a local server an identity to check: stdio has no authentication by construction and inherits the user's whole session, while a loopback HTTP server is callable by every process on the box and — via DNS rebinding, because browsers compare domain names rather than resolved IPs — by any web page the user visits, unless it validates the Origin header the spec makes a MUST. Public reporting says the TypeScript, Python and Go SDKs shipped that check off by default. Adobe's 10 November 2026 MAX preview of local MCP for Photoshop, Illustrator, Premiere and After Effects (private beta early 2027) points at what comes next, where the target is the open document rather than a path you can allowlist. The controls that work are an inventory, a pinned allowlist, a transport rule and a one-curl acceptance check.