Computer use agents are a last resort, and that's fine

What a computer use agent is, in one loop

On OSWorld — 369 real tasks on real operating systems — the human baseline is 72.36%. In October 2024, the best computer use model scored 14.9% on the screenshot-only subset. By September 2025, Anthropic reported Claude Sonnet 4.5 at 61.4% on OSWorld-Verified.

A computer use agent is a model that operates a computer the way a person does: it receives a screenshot, decides on a single mouse or keyboard action, executes it through a driver outside the model, receives a new screenshot, and repeats until the task completes or a limit stops it. No API. No DOM contract. Pixels in, coordinates out.

That is the entire architectural claim. Everything interesting about these systems — the cost, the failure modes, the security problem — falls out of the fact that the model's only sensor is a bitmap and its only actuator is a coordinate pair.

Anthropic's launch post called the tool "at times cumbersome and error-prone" and named scrolling, dragging and zooming as the unreliable parts. Day one. Vendor. Flagship capability. Remember that when someone demos it to you.

The loop, and where each step breaks

Three documented failure modes, one per step.

Stale pixels. The screenshot is a point-in-time snapshot. Between capture and click, a modal opens, a spinner resolves, an ad loads, a list reorders — and the coordinate the model chose now points at something else. Anthropic's docs tell you to hold the screen at XGA (1024×768) and warn that higher resolutions get resized, degrading coordinate accuracy. The fix for precision is a smaller, more stable canvas. Not a bigger one.

Unbounded loops. The model cannot tell that it is not making progress, so it retries the same click forever. This is the most common production failure for any agent, not just this class. The fix is a hard step cap and a wall-clock timeout enforced in the driver, outside the model, where no amount of reasoning can talk its way past it. Same principle I've written about in building one bounded agent — the bound is the product.

Injected instructions. The screenshot contains text. The model reads text. There is no channel separation between "the page I am looking at" and "the instructions I am following," and there is no fix for that at the model layer.

Who ships one, and what they actually control

The control surface is narrowing, not widening. Anthropic opened with the whole desktop in October 2024. OpenAI shipped a browser three months later. Google scoped its model to browser control and said so in the announcement. The vendors learned that most valuable work happens inside a browser and that a browser is vastly easier to contain than an OS.

The benchmark column does not compare across rows. OSWorld-Verified is a corrected revision of the original task set, and the 14.9% figure was on a screenshot-only subset. Anyone quoting "14.9% to 61.4%" as a clean year-over-year curve — including me, two sections ago — is compressing three different measurement regimes. The direction is real. The slope is not something you can extrapolate.

What forty steps actually cost

Anthropic's vision docs give the token estimate as (width × height) / 750. A 1024×768 screenshot is 786,432 pixels, so roughly 1,049 tokens per screenshot — per image, per request. Keep the last three screenshots in context for visual continuity and you are carrying ~3,150 image tokens before a single word of task description, tool definition, or action history.

Then add the history. Step 30 of a 40-step task sends the system prompt, the tool schema, the task, and 29 prior observations and actions, so input tokens per step climb through the run. Cost is roughly quadratic in steps, not linear. That is the part people skip.

Here is the scenario — a 40-step browser task, about 12,000 input tokens per step averaged across the run, short actions out, run a thousand times a month. Change the model and the sliders to match yours.

Two levers move this number. Prompt caching on the static prefix — system prompt, tool definitions, task — is the one to reach for first, because it cuts the repeated portion without touching what the model can see. Dropping older screenshots while keeping their text summaries cuts the image load harder, and it will cost you tasks that depend on remembering what a screen looked like six steps ago. Pay that price deliberately, on a per-workflow basis, not globally.

The lever that does not work is a bigger screen. More pixels means more tokens and, per Anthropic's own guidance, worse coordinate accuracy after resizing. You pay twice to get less.

Prompt injection is structural, not a bug in the queue

Anthropic published red-team numbers for Claude for Chrome in August 2025: autonomous browsing without mitigations showed a 23.6% attack success rate, reduced to 11.2% with their safety mitigations in place.

Read the second number again. Mitigated, shipped to a controlled preview of 1,000 users, and one attack in nine still lands.

This is not a tuning problem. The model's input is a rendered page, and a rendered page is authored by whoever controls the server. An invisible div, white text on white, a comment in a support ticket, alt text on an image — all of it becomes model input, indistinguishable in kind from your instruction. OpenAI's computer use guide returns safety checks the developer must explicitly acknowledge before the loop continues. Anthropic's docs tell you not to give the model access to sensitive accounts at all. Both vendors are pushing the decision back to you, and they are right to.

So the mitigation that works is not in the prompt. It is in the blast radius: what credentials exist in that session, what the network allows, and what actions require a human. An agent in a container with no saved cookies and no outbound access except two allowlisted domains can be injected all day and achieve nothing.

What I would not use

I would not drive a GUI where a documented API or an MCP server exists. Pixel control is the most expensive, slowest and least verifiable way to make a computer do something. An API call has a schema, an error code and an idempotency key. A click has a coordinate and hope.

I would not use a browser extension running in my real, logged-in profile. Claude for Chrome's appeal is exactly its risk: it has your sessions. That is the configuration where an 11.2% injection success rate turns into an incident, because the agent is already authenticated to everything you are. Run the loop in a container with a fresh profile — Anthropic ships a reference implementation for precisely this, a Docker image with a virtual display and VNC so you can watch it. If you want the longer argument about what "self-hosted" buys you here, I've split that into its two meanings.

And I would not give one spend authority. I took Metadata.io from $0 to $15M ARR building software that moved real ad budgets automatically, and I hold six patents in AI-driven marketing, so I am not squeamish about machines spending money. I am squeamish about a machine that reads its instructions off a bitmap spending money. Different systems. Metered, bounded, per-task billing is a well-understood pattern with documented implementations — here is one worked example of a metered billing build — and the gate belongs between the agent's output and the payment rail, not inside the agent's reasoning.

For structured action on a web app with no API, start with the DOM. Projects like browser-use operate on the accessibility tree and element handles rather than pixels, which gives you stable selectors, cheaper context, and errors you can assert on. Pixels are the fallback for when the DOM lies: canvas apps, remote desktops, Citrix, legacy Windows software, PDFs rendered into viewers.

Where this is genuinely the right tool

All three share one property: no alternative interface exists.

Notice what is absent. Anything high-volume, anything touching money, anything where the same operation is available as an HTTP call. Those are integration problems wearing an agent costume.

Computer use is the last mile. The model that can drive a GUI is impressive and the OSWorld curve from 2024 to 2025 is steep, but a 61.4% task success rate means roughly two in five attempts fail on tasks a human completes seven times in ten — and in production the cost of a failed attempt is not a benchmark point, it is a half-submitted form in someone else's system. Build the bound before you build the capability. The capability is rented from a vendor and improves monthly. The bound is yours and nobody ships it for you.