AgentOps: Deploy & Operate

Durable State & Resumability

Make the agent loop a durable computation — event-sourced history, journal-before-effect, and resume that replays rather than re-derives, so a crash or redeploy never restarts a half-done task.

Concurrency, Queues & Scaling

Agents are batch jobs, not requests: a queue with leased workers, per-tenant concurrency caps, journal-as-state for horizontal scale, and bounded fan-out are what survive production load.

Idempotency, Retries & Side-Effect Safety

Four stacked retry sources mean every write tool will fire twice unless you construct exactly-once with intent-derived idempotency keys, failure classification, and a durable side-effect ledger.

Cost Control at the Loop Level

Agent cost is unbounded by default; treat the per-task token/step/dollar ceiling as a fail-closed circuit breaker, then tune model cascades, prompt and tool caching, and early-exit against a quality metric.

Rollout, Versioning & Pinning

Behavior is the (model, prompt, tools) triple; pin it to dated snapshots, stamp it on every run, and promote new versions only through shadow/canary plus an eval gate with instant config-flip rollback.

Feature flags for agents

Flags scoped to prompts, models, tools, and policies — what to gate, how to roll, and why "off by default" is a non-negotiable for agent flags.

Kill switches

A button that stops a running agent fleet — what it must actually stop (in-flight calls, queued work, scheduled retries), and how to test it before you need it.

Incident Response & Runaway Containment

A runaway agent fails open and keeps acting; detect from rate and progress, contain with in-loop fail-closed kill switches the resume path respects, rely on pre-installed blast-radius bounds, and turn every incident into a regression test.

Rate Limits & Provider Capacity

A 429 is a capacity contract, not a transient error, and retrying it turns a shortfall into an outage; agents burn tokens-per-minute quadratically, so the fix is a shared admission-control bucket, deliberate load shedding, and treating cross-provider failover as a behavior change your evals must cover.

Model Deprecation & Migration

A model ID is a dependency with an expiry date you cannot vendor: notice floors of 60 days or less, why the swap is a re-qualification rather than a string replacement, silent platform auto-upgrades as the worst failure mode, and the generated inventory plus warm candidate lane that turn a retirement into a one-day operation.

Self-Hosted Inference for Agents

Leaving the provider API changes the currency from tokens to KV-cache bytes: capacity is concurrent sequences times context length, prefix-cache hit rate is a routing problem rather than an engine one, autoscaling does not work at agent timescales, and the break-even is a utilisation number.

Multi-Tenancy for Agents

An agent adds five stores your row-level policy never touched — prompt cache, semantic cache, vector index, memory and the rate-limit pool — and only the semantic cache can hand one tenant another tenant’s answer.

SLOs & Error Budgets for Agents

Correctness fails every test an SLI must pass, so budget the mechanical indicators you can compute deterministically, run a second harm budget denominated in actions taken rather than requests served, and demote judge-scored quality to a control chart that never pages.

Graceful Degradation & Fallback

The fallback is a different agent — different tool dialect, context ceiling and refusal profile — running your most traffic down your least-tested path under thresholds calibrated for a model that stopped answering: decide what to shed before what to swap, fail closed on side effects, and make degraded mode a named state with an exit.

Exiting a Managed Agent Runtime

Maintenance mode promises existing workloads keep running, which is exactly what makes teams wait — the real deadline is the day the frozen model catalog stops carrying a model you need, and meanwhile the loop everyone assumes is the hard part ports in a sprint while the conversation state nobody inventoried is the migration: prove the export on day one, dual-write, then port code against a backlog that has stopped growing.

Third-Party Tool Drift

A tool description is part of your prompt and someone else owns the text, so behaviour changes with no commit and no error — the loud structural breaks are the harmless kind, while a reworded docstring moves tool-selection rates silently: snapshot the catalog, stamp its hash on every trace, and absorb the change in a facade rather than the prompt.

Scheduled & Triggered Agents

With no user present, ask becomes abstain, retry becomes reconcile, and the default failure is silence rather than an error — so put a dead-man’s switch on every schedule, end each firing in an acted / no-op / blocked verdict, and rate-limit the notification channel independently of the agent’s own judgement.

Load-Testing an Agent System

The first decision is what to do about side effects, and every answer changes the measurement — mocked tools delete the seconds of real latency that dominate a trajectory. Size the run from Little’s Law, generate sampled tasks rather than one repeated prompt, grade quality under load because degradation is silent, and report the concurrency where success starts falling plus which dependency returned the first 429.