Playbooks / Coding & Computer-Use Agents
Coding & Computer-Use Agents
Agents that read code, write code, run tools, and drive a computer — patterns, harnesses, and pitfalls.
- Coding Agent ArchitectureThe localize-edit-verify loop that makes a coding agent more than a code generator: the agent-computer interface, why agentic beats pipeline coding, and where the loop fails.
- Repo Navigation & Code ContextCursor removed its semantic codebase index during 2026 and Claude Code never had one, and the reason is not that embeddings stopped working: a May 2026 study (arXiv:2605.15184) held the retrievers fixed and found grep beat vector search on all ten harness-and-model pairs when hits came back inline, while vector won five of ten when the same hits were written to files the agent had to open. So the delivery channel decides it, not the retriever — and once search is free the bill moves to reading, where a grep returning 12 hits costs ~700 tokens and opening the three files they sit in costs ~24,000. Covers what an index still buys (the query whose name nobody wrote, bad legacy naming, latency on a huge tree), the symbol tier via LSP or tree-sitter, and why confident wrong localization is the expensive failure.
- Patch Generation & Test-Driven LoopsGreen means "nothing that ran failed", and the distance to "correct" is now measured: on SWE-bench Verified 7.8% of patches counted as correct while failing the developers' own tests, differential testing put 29.6% of plausible patches at behaviour differing from the ground truth, and adversarial test generation strengthened 50.2% of instances. Structured diffs, repro-first, regression baselines and the three honest liars — plus the step most loops are missing: measure the suite's discriminating power with mutation on the touched files, and make the agent ship a test that goes red on the pre-patch code.
- Computer-Use & GUI AgentsPer-step reliability compounds, so the job is fewer steps that must be right, not smarter ones: 0.97^40 completes 29.6% of tasks and 0.995^40 completes 81.8%, a 2.8x swing, and both levers sit in the harness. Covers hybrid pixel and accessibility-tree grounding, the zoom-before-prompt-engineering order the vendor docs now recommend, why a GUI score without its step limit, resolution and tool permissions is not a measurement, the step-21 image-limit cliff, and the five reliability moves that beat picking a stronger model.
- Browser agentsDriving a real browser as a tool — DOM versus pixel observation, login + auth state, the well-trodden failure modes, and when to step up to a full GUI agent.
- IDE agentsCoding agents that live in the editor — the loop is the same as a CLI coding agent, but the interaction surface, undo expectations, and trust threshold are all different.
- Sandboxing & Safe ExecutionFirecracker's first two CVEs both landed in 2026 and neither is the kernel escape the threat model assumes — one crashes the VMM from guest root on an opt-in transport, the other needs host write access first — while the year's one confirmed guest-to-host root escape sits in KVM, below every vendor you could switch to. So audit reachability instead: a seven-line probe run inside the real sandbox image, the IMDS hop-limit hardening that stops working in two of three ECS network modes, and an egress policy written on method-and-destination because the 3,022-package RubyGems campaign went entirely to allowed hosts.
- Evaluating Coding AgentsSWE-bench’s tests have been shown both to accept wrong patches and to reject correct ones, so a green suite is necessary but not sufficient: mine tasks from your own merged PRs, hide the tests, grade the diff a reviewer would merge, report pass^k rather than pass@k, and track cost per mergeable diff in CI.
- Code Review AgentsA review bot lives or dies on precision, not recall: diff-anchored context, an adversarial gate that drops any finding without a concrete failure scenario, a hard comment budget ranked worst-first, and acted-upon rate as the one production metric.
- Large-Scale Migration AgentsGeneration went to zero and human review did not, so the deliverable is evidence rather than patches: build the oracle first, batch by verifiability instead of by directory, hand the mechanical head to a codemod and only the tail to the model, and let a hundred-file pilot’s unedited-merge rate decide whether the project is viable.
- Debugging & Triage AgentsAn agent that reads a stack trace and emits a diff has pattern-matched, not debugged — make the failing test the deliverable and the eval criterion becomes objective, the spend moves from generation to observation, and "cannot reproduce" becomes a result you can trust.
- Background Coding AgentsAn agent that opens twelve pull requests a day adds nothing if your team merges four — detaching from the editor moves the bottleneck to review, so every decision is either about making a run self-verifying or about keeping the queue short enough that the work lands.
- Test-Generation AgentsAn agent that writes tests from your code infers the spec from the implementation, so wherever the code is wrong the test now certifies the bug and blocks the fix — coverage cannot see this, mutation score can, and the only jobs worth dispatching are the ones where you can name the oracle in a sentence.
- Dependency Upgrade AgentsBumping a version has been automated since 2017 and is worth nothing — the backlog exists because nobody will merge an upgrade they cannot vouch for, so what you are building is an evidence policy: a green suite proves least about the changed defaults that break silently, and correct triage beats upgrades merged.
- Documentation AgentsDocumentation is the one artefact in the repo with no oracle, and a wrong paragraph gets believed for years — so split the corpus by what a machine can falsify, give the agent unsupervised authority over derived reference and executable prose only, make deletion a first-class output, and report freshness rather than pages written.
- Design-to-Code AgentsA screenshot cannot say "this is the Button component", so a pixel-faithful generator rebuilds one, hard-codes the hex instead of the token, and passes every visual review — judge the agent on component reuse rate and token adherence instead, hand it the typed component API and the design-to-import mapping before the picture, and if a ten-screen pilot comes in under seventy per cent reuse you have a design-system problem the agent will only scale.
- Vulnerability Remediation AgentsGenerating the patch is the cheap half — about a quarter of model-written patches for post-cutoff CVEs fix the bug without changing behaviour, and more than forty per cent of the ones your validation calls correct fail once you test mutated exploits rather than the one you were handed, so the deliverable is a reproduction that fails before and passes after, on a queue triaged by reachability instead of scanner severity.
- Spec-Driven Development with Coding AgentsThe plan, the tests, the code and the summary all descend from one reading of the task, so when that reading is wrong every artefact agrees and the review passes — a spec earns its place only insofar as it comes from outside that loop, which means human-authored observable criteria with an explicit non-goals list, a check id or a named human beside each one, and spec and behaviour changing in the same pull request.
- Performance-Optimization AgentsEvery other coding agent gets a verifier that cannot lie; this one gets a stopwatch — so forty candidate patches scored on one timing run each will keep noise with near-certainty. Build the harness first (confidence intervals, a published minimum detectable effect, a fixed repetition budget), gate on the full test suite rather than folding correctness into the score, and hand the agent a profile instead of a repository.
- Infrastructure-as-Code AgentsEvery other coding agent has to build its own oracle; this one is handed terraform plan for free — so the deliverable is a plan whose every line falls into a class you already agreed to auto-apply, the review job becomes classifying the plan rather than reading the diff, and the engineering is all in the four things a plan is silent about: unknown values, server-side behaviour, replacements announced in the same tone as everything else, and whatever is not in state.
- CI Repair AgentsHanded a reproduction on a plate, this agent’s real job is the classification nobody runs: caused by this diff, pre-existing on the base, infrastructure, or non-deterministic — prove causation by running the failing check against the merge base before pushing, spend at most one re-run, never touch what a test asserts, and page yourself on wrong-push rate rather than builds turned green.
- Database-Migration AgentsA model writes correct DDL first time, which is why this is the coding-agent task most likely to take production down: the statement is fine and the sequence is wrong, and CI runs it against an empty table with nobody else connected. Make the deliverable an ordered expand–backfill–contract sequence the currently-deployed code still fits, default the agent to additive-only changes, emit the lock_timeout preamble every time, and grade it on lock duration measured by a shadow apply against a production-shaped clone under load — the agent authors, a controlled runner applies.
- Notebook & Data-Science AgentsThe file on disk is not the program that produced the answer — the kernel is, and nothing writes it down, so “it ran” is unfalsifiable until a cold restart-and-run-all executed by your harness says otherwise. Make that gate the definition of done and hand it to the agent as a tool, feed it schema cards instead of printed dataframes, remember the dangerous boundary is the warehouse credential rather than the sandbox, and grade the number and the method rather than whether the code ran.
- Mobile & Native App AgentsA coding agent’s advantage is being wrong twenty times an hour, and a clean iOS or Android build spends that budget before lunch — so the fix is not a faster build but a codebase split into a fast core the agent iterates in and a slow shell the full build gates once per candidate. Pin the simulator or every red is ambiguous, make committed snapshot references the contract the agent may propose but never accept, and rank the backlog by full builds per attempt.
- Accessibility Remediation AgentsGive an agent a scanner score and it builds you an accessibility overlay inside your own repository, because the cheapest way to silence a rule is an ARIA attribute that lies — and automated testing reaches roughly half the failures by count and almost none of the ones that block a journey. Make the unit of work a keyboard-only user journey, fix at the design system rather than the call site, and require every pull request to carry a before-and-after focus trace as its proof.
- Dead-Code & Feature-Flag Removal AgentsDeletion is the one coding-agent task where a green suite proves nothing — it passes for exactly the reason the code looked dead — and coverage rises when you delete untested code, so the obvious objective rewards removing what you understand least. Static analysis proposes and never decides; the evidence has to be runtime reachability over a window set by the business calendar, not a round thirty days. A flag evaluated a million times returning false is not a flag never evaluated, a kill switch reads exactly like a stale flag to every heuristic, and the production value is usually the opposite of the code default.
- Merge Queues for Agent-Authored ChangesReview is not the bottleneck — 31% more PRs now merge with no review at all, and the incidents-to-PR ratio tripled, so the system absorbed the volume by routing around its own gate. The serialized path to main is the gate that is left, and batching (the standard fix) inverts under agent load: at a 20% per-PR queue-failure rate a batch of ten passes 10.7% of the time and costs you a full CI run per merge. Size batches to measured p, insist on bisection, and rebase-and-verify against the queue head before anything enters.
- Release & Publishing AgentsEvery other coding-agent task is revertible; a published release is not — npm’s unpublish window is 72 hours, the version number can never be reused, and the lockfiles that already resolved it are permanent. So the design centre is the signing boundary, not the automation: let the agent prepare the bump, the notes and the dry run, and arrange that no long-lived publish credential exists for it to hold. Provenance attests where an artefact was built, not that a human agreed to ship it — so if your agent can push to the ref your trusted publisher builds from, the attestation is valid and says the wrong thing. Measure time-to-supersede, and make the forward-only fix path a tool the agent can call.
- Eval-Authoring AgentsWhat makes test generation work is a free oracle, and agent evaluation has none — so the only durable split is agent-generates-situations, something-else-owns-the-answer, and a set where the same model family writes both case and grader will report improvement forever. Mine traces for named failure shapes before generating, rank every case by the strongest oracle it can reach, quota a fifth of the set as unsatisfiable with credit for refusing, and keep only cases that separate two systems you already know differ.
- Secret-Scanning & Rotation AgentsDetection is the solved half and you are paying for it twice — over 64% of credentials confirmed valid in 2022 were still valid at a 2026 retest, so the deliverable is a credential proven dead, with a receipt. Proving it requires using it, which forces the architecture: the agent sees fingerprints while a separate verifier service holds the secret, and the agent may create credentials but never revoke them.
- Web-Crawling & Site-Reading AgentsWhat gets you blocked is never total volume — it is concurrency against one origin and the share of reads that are repeats, and neither is something a prompt can fix: build the fetch service first, with a fleet-global per-origin budget at one concurrent request, a shared cache that sends conditional requests, and a route table that takes the bulk feed your organisation already pays for instead of the retail page that returns 200.
- Agents on Developer WorkstationsOn a laptop every control sits inside the trust domain of the person it constrains — they are root. Local sandboxes are now deny-by-default on the network, so the axis that still separates them is whether the agent can read the token at all.
- Data-Pipeline Agents (dbt & Airflow)A transformation's correctness lives in the rows it produced, so the test suite is not the oracle — the row-level diff against production state is. Give the agent a deferred build, gate the merge on a delta it predicted in advance, and distrust state:modified, which dbt's own docs admit cannot see a var or env_var change and hope only to "reach eventual consistency". Airflow's tasks test skips dependency checks and DB state but runs the operator for real — there is no --dry-run — and only the Git DAG bundle versions, so a mid-run merge on a local or S3 bundle swaps the code under a live run. Set autonomy from Spider 2.0-DBT: the best frontier models on the reference harness solve 35-40% of 68 repository-level tasks.
- Flaky-Test Triage AgentsProving a test is flaky costs one rerun; proving one is healthy costs 59 consecutive passes at a 5% flake rate and 299 at 1% — so quarantine entry can be automatic and quarantine exit can only ever be a deadline with an owner. Key history to a stable test identity rather than a name, keep the quarantined test running and reporting, and forbid the repair that actually works: retrying a failure three times takes a 1-in-20 failure rate to roughly 1 in 8,000 for about 5% more runtime, and hides a genuine regression exactly as well.