Computer-Use & GUI Agents

8 min read

U4
Playbook · Coding & Computer-Use Agents

A GUI agent trades a clean API for a screenshot and a mouse — and pays for it every single step.

Per-step reliability compounds, so the engineering problem in GUI control is not making each step smarter but reducing how many steps have to be right: at 97% per step a 40-step task succeeds 29.6% of the time, and at 99.5% it succeeds 81.8% of the time — a 2.8× swing you buy with addressable targets, keyboard actions and an explicit check after every action, not with a better model. Everything below is in service of that one number.

Step 1Grounding is the bottleneck: knowing what to click vs. where it is.

Two distinct skills, and the second is where agents fail. Semantic grounding (“I should click Submit”) is comparatively easy; spatial grounding — producing the pixel coordinate of that button on this rendering, at this resolution, with this theme — is where OSWorld-class benchmarks show the gap. Dedicated grounding suites exist precisely because element localisation, not planning, is the dominant error: the agent knows the right action and clicks ten pixels off a moved button.

The vendor documentation now says this out loud in the shape of its own remedies. Anthropic's computer-use reference recommends 1024×768 or 1280×720 for desktop tasks and 1280×800 or 1366×768 for web, warns against anything above 1920×1080, and when clicks miss advises enabling zoom — a member of the current toolset that re-captures a region at full resolution — before anything else. The order of that advice is the finding: resolution at the point of the click is a bigger lever than prompt engineering around the click. Note also that the toolset's higher ceiling is not available everywhere; which tool version you get depends on the hosting platform, which is a portability problem in its own right.

Step 2Pixel grounding is universal; DOM and accessibility grounding are addressable — use both.

A screenshot works on a native app, a Citrix session, a game — anything — but coordinates are fragile to layout, DPI and scroll. The DOM or accessibility tree gives stable, addressable element handles, and only exists for web and instrumented apps. The strong pattern is hybrid: prefer the accessibility tree where it exists for a robust target id, fall back to pixel grounding where it does not, and cross-check the screenshot against the tree so a stale DOM does not point the click at an element that visually moved.

# perceive -> ground -> act -> verify, one GUI step
shot = screen.capture()
tree = a11y.tree()                       # stable handles where available
tgt  = agent.ground(goal, shot, tree)    # element id OR (x, y)
ui.click(tgt); ui.type(text)
assert screen.capture() != shot          # did anything happen at all?

The action space is small — click, double-click, type, scroll, key, drag, and the waits — but the failure space is huge: a click that landed on nothing, a modal that stole focus, a page still loading. Every action needs an explicit did-it-take-effect check, and the documentation is blunt about why: “Claude sometimes acts as if its actions succeeded without explicitly checking their results.” An agent that skips verification confidently builds twenty steps on top of a no-op.

Coordinates come back in screenshot pixels with the origin at the top left, so if you downscale before sending you must scale the model's answer back up — and a rounding error there is a click that lands near the button rather than on it. That is why the resolution knob is not purely an economic one, even though it dominates the bill.

Step 3The arithmetic that decides everything: reliability compounds.

This is the load-bearing calculation of the whole discipline, and it is worth doing on your own numbers before choosing anything else. A GUI task is a serial chain; a task succeeds only if every step does.

# success = (per-step reliability) ^ (number of steps)

0.97  ^ 40  = 0.296      # 97.0% per step -> 29.6% of tasks complete
0.99  ^ 40  = 0.669      # 99.0% per step -> 66.9%
0.995 ^ 40  = 0.818      # 99.5% per step -> 81.8%
0.97  ^ 20  = 0.544      # 97.0% per step, half the steps -> 54.4%

# 97.0% -> 99.5% per step:  2.8x more completed tasks
# 40 steps -> 20 steps:     1.8x more completed tasks
# Both levers are in the harness, not in the model.

Three consequences follow. First, a task that is a three-line script through an API becomes a 40-step GUI sequence, and the reliability penalty is multiplicative rather than additive — which is the structural reason GUI control is a last resort rather than a default. Second, the two levers that matter are per-step reliability and step count, and a better model is only one of several ways to move the first. Third, because you are working against an exponent, the marginal value of a point of reliability rises as reliability does: one percentage point is worth 15 points of task completion at 97% (29.6% to 44.6%) and 33 points at 99% (66.9% to 100%), so the expensive last point is the valuable one. Anything that shaves a step — a keyboard shortcut instead of a menu traversal, a direct URL instead of three clicks, a bulk operation instead of a loop — pays twice, once in cost and once in the exponent.

Step 4What a GUI benchmark number does and does not tell you.

OSWorld and its successors are the reference suites, and their headline numbers have moved fast enough that quoting one is a dated statement rather than a durable one. The original OSWorld paper reported a human baseline of 72.36%, with the best model of its day near 12%. Top reported scores on OSWorld-Verified have since passed that human baseline, while the longer-horizon OSWorld 2.0 suite scores far lower — third-party trackers put the best official entry's full-completion rate in the mid-40s.

The useful part is not the level but the spread. As of late 2026 third-party leaderboards disagree with each other about which system leads OSWorld-Verified, on the same date, by more than ten points. That disagreement is not sloppiness; it is the benchmark reporting a different quantity under each configuration. Self-reported rows vary in maximum step count, screen resolution, OS image and tool permissions, and OSWorld 2.0 reports both a strict full-completion rate and a partial-credit score that can differ by tens of points on the same run.

A GUI agent score without its step limit, its screenshot resolution and its tool permissions is not a measurement of the agent. Treat the triple as part of the number, demand it of any vendor claim, and when you build your own suite report it alongside your result — the same discipline evaluating coding agents needs, and for the same reason. See also the benchmark landscape for how the suites relate.

Step 5Reliability engineering: make fewer steps, and make each one repeatable.

Because steps fail and the loop is long, robust GUI agents borrow from distributed systems rather than from prompt engineering. Five moves, in descending order of how much they buy.

  • Prefer the keyboard. A shortcut, a tab order or a typed path is addressable and resolution-independent; a pixel-targeted click on a menu item is neither. This is the cheapest way to raise per-step reliability and it removes steps at the same time.
  • Make actions idempotent and checkpoint between them. Re-issuing a click should not double-submit, and a failure at step 30 should not restart from zero. The patterns are the ordinary ones — idempotency keys and bounded retries — applied to a UI instead of an API.
  • Verify explicitly, cheaply. A full screenshot comparison after every action is both expensive and ambiguous. Prefer a narrow check: the element that should now exist, the field that should now hold a value, a zoom on the region you just touched rather than a re-read of the frame.
  • Detect stuck rather than waiting it out. The same screenshot N turns running is the signal; break the loop, do not keep paying for it. Long GUI runs fail by looping far more often than by crashing, and a stuck detector is the difference between a failed task and an unbounded bill.
  • Watch the step-21 cliff. Once a request carries more than 20 images the documented per-side limit tightens to 2,000 pixels, so a full-retention loop silently changes its own observation contract partway through a long task. This reads as an unexplained mid-task accuracy drop and is a retention-policy bug, not a model one.

Notice what is absent from that list: “use the strongest model.” The reference compares the earlier tool version's models directly and finds Claude Sonnet 4.6 “more mechanically precise at clicking than Claude Opus 4.6,” with Claude Opus 4.7 “roughly comparable to Sonnet 4.6.” On this surface the best reasoner is not automatically the best actuator, which is one more reason to measure your own configuration instead of inheriting a ranking.

Step 6When NOT to use a GUI agent.

GUI control is the interface of last resort: use it only when there is no API, no CLI and no scriptable path — legacy enterprise apps, third-party SaaS without integrations, human-only workflows. Anywhere a stable programmatic interface exists it is faster, cheaper, and an order of magnitude more reliable, because an API call does not have a per-step grounding probability at all. A pixel is the most expensive API you will ever call.

There is also a middle option that gets skipped. For web targets specifically, driving the DOM through a browser-automation layer gives you addressable elements without the screenshot loop, at the cost of working only where the page cooperates; browser agents covers that path and browser-agent failure modes covers where it breaks. Reach for full pixel-level computer use when the target is a native or remote application that exposes nothing else.

What to do. Before writing any prompt, count the steps your task needs and multiply out your current per-step success rate — if the product is under 50%, work on step count and grounding, not on the model. Then, in order: pin the capture resolution to the documented recommendation rather than the display's; replace every mouse action you can with a keyboard one; add a narrow verification after each action and a stuck detector over the last N frames; make the history append-only so the per-step perception cost stays linear; and record the step limit, resolution and tool version next to every result so your numbers mean something next month. For anything irreversible at the end of a long chain, do not rely on the agent's own verification — put a computed preview or an undo path behind it, because the one thing a 40-step chain reliably produces is a state nobody predicted.