Build a working computer-use agent loop in one Python file

What this script does when you run it

One Python file. It screenshots a browser page, sends the PNG to Claude with the computer_20250124 tool, receives back something like {"action": "left_click", "coordinate": [412, 331]}, and executes that click with Playwright — then screenshots again and repeats until the task is done or a step counter kills it. Budget about 20 minutes. Most of that is playwright install chromium downloading a browser.

You need Python 3.10 or newer and an Anthropic API key with billing enabled, from console.anthropic.com/settings/keys; computer use is a beta feature on the paid API. Then pip install anthropic playwright, then playwright install chromium (install docs). No VM, no Docker, no X server — Chromium runs headless and the screenshots come from Playwright, not a desktop.

The loop is four moving parts, and one of them is a counter

Pixels in, a coordinate out, an actuator that performs the coordinate, and a bound on how many times that can happen. The model never touches your machine. It emits JSON. Your code decides whether to honour it.

Two things in that diagram matter more than the rest.

First, the coordinate space. You declare display_width_px and display_height_px when you define the tool, and the model returns coordinates inside that space. Send an image at a different size than the numbers you declared and every click lands in the wrong place with no error at all. Anthropic's docs recommend keeping the display at or below XGA (1024×768) or WXGA (1280×800); larger screenshots get resized and coordinate accuracy degrades.

Second, the step cap is not in the model. It is a for loop in your code. Unbounded iteration is the standard production failure for this architecture — the model re-screenshots, decides it is almost there, acts again, and never emits a stop. The only fix that holds is a counter enforced outside the model. That is why MAX_STEPS is a constant in the file below and not a line in the prompt.

The target page

Save this as page.html in an empty directory. It posts to example.com, a reserved documentation domain, so a successful submit produces a visible, checkable navigation.

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>Example subscribe form</title>
</head>
<body>
  <h1>Subscribe to the Example newsletter</h1>
  <form action="https://example.com/" method="get">
    <p><label>Email address
      <input id="email" name="email" type="email" size="40" placeholder="you@example.com">
    </label></p>
    <p><button type="submit">Subscribe</button></p>
  </form>
</body>
</html>

The task given to the agent: type ada@example.com into the field, click Subscribe, then report the heading it lands on. Clicking that button navigates to https://example.com/?email=ada%40example.com, whose heading is "Example Domain". That string is the proof the loop closed.

The agent

Save as agent.py, same directory. This is the whole thing.

#!/usr/bin/env python3
"""Minimal computer-use agent.

Loop: screenshot -> Claude (computer tool) -> action JSON -> Playwright -> screenshot.
"""
import base64
import os
import time

from anthropic import Anthropic
from playwright.sync_api import sync_playwright

WIDTH, HEIGHT = 1024, 768       # must match the tool's display_* values exactly
MAX_STEPS = 12                  # hard stop, enforced outside the model
KEEP_SCREENSHOTS = 3            # older images are replaced with a text stub
MODEL = "claude-sonnet-4-5"
BETA_FLAG = "computer-use-2025-01-24"

GOAL = (
    "Type ada@example.com into the email field on this page, then click the "
    "Subscribe button. Once the page has finished navigating, reply with the "
    "exact text of the main heading you can see. Do nothing else."
)

# xdotool key names (what the computer tool emits) -> Playwright key names
KEY_ALIASES = {
    "Return": "Enter", "KP_Enter": "Enter", "BackSpace": "Backspace",
    "Escape": "Escape", "Page_Down": "PageDown", "Page_Up": "PageUp",
    "Next": "PageDown", "Prior": "PageUp",
    "ctrl": "Control", "super": "Meta", "alt": "Alt", "shift": "Shift",
}


def to_playwright_key(combo):
    return "+".join(KEY_ALIASES.get(p, p) for p in combo.split("+"))


def shot(page):
    return base64.b64encode(page.screenshot(type="png")).decode()


def execute(page, args):
    """Perform one computer-tool action and return the resulting frame."""
    action = args["action"]
    x, y = args.get("coordinate", (0, 0))

    if action == "screenshot":
        pass
    elif action == "mouse_move":
        page.mouse.move(x, y)
    elif action in ("left_click", "right_click", "middle_click",
                    "double_click", "triple_click"):
        button = {"right_click": "right", "middle_click": "middle"}.get(action, "left")
        count = {"double_click": 2, "triple_click": 3}.get(action, 1)
        page.mouse.click(x, y, button=button, click_count=count)
    elif action == "type":
        page.keyboard.type(args["text"], delay=20)
    elif action == "key":
        page.keyboard.press(to_playwright_key(args["text"]))
    elif action == "scroll":
        step = int(args.get("scroll_amount", 3)) * 100
        dx, dy = {"down": (0, step), "up": (0, -step),
                  "right": (step, 0), "left": (-step, 0)}[args["scroll_direction"]]
        page.mouse.move(x, y)
        page.mouse.wheel(dx, dy)
    elif action == "wait":
        time.sleep(min(float(args.get("duration", 1)), 3))
    else:
        return {"text": "unsupported action: " + action}

    page.wait_for_timeout(500)   # let the DOM settle before the next frame
    return {"image": shot(page)}


def prune_images(messages, keep):
    """Keep only the most recent `keep` screenshots in the transcript."""
    seen = 0
    for message in reversed(messages):
        content = message.get("content") if isinstance(message, dict) else None
        if not isinstance(content, list):
            continue
        for block in content:
            if not isinstance(block, dict) or block.get("type") != "tool_result":
                continue
            for i, part in enumerate(block.get("content", [])):
                if isinstance(part, dict) and part.get("type") == "image":
                    seen += 1
                    if seen > keep:
                        block["content"][i] = {"type": "text", "text": "[screenshot pruned]"}


def main():
    client = Anthropic()  # reads ANTHROPIC_API_KEY from the environment
    tools = [{
        "type": "computer_20250124",
        "name": "computer",
        "display_width_px": WIDTH,
        "display_height_px": HEIGHT,
    }]
    start_url = "file://" + os.path.abspath("page.html")

    with sync_playwright() as pw:
        browser = pw.chromium.launch(headless=True)
        page = browser.new_page(
            viewport={"width": WIDTH, "height": HEIGHT},
            device_scale_factor=1,   # 1 CSS pixel == 1 image pixel
        )
        page.goto(start_url)

        messages = [{"role": "user", "content": [
            {"type": "text", "text": GOAL},
            {"type": "image", "source": {"type": "base64",
                                         "media_type": "image/png",
                                         "data": shot(page)}},
        ]}]

        for step in range(1, MAX_STEPS + 1):
            response = client.beta.messages.create(
                model=MODEL,
                max_tokens=1024,
                tools=tools,
                betas=[BETA_FLAG],
                messages=messages,
            )
            print("[%d] usage in=%d out=%d" % (
                step, response.usage.input_tokens, response.usage.output_tokens))

            for block in response.content:
                if block.type == "text" and block.text.strip():
                    print("[%d] say %s" % (step, block.text.strip()))

            calls = [b for b in response.content if b.type == "tool_use"]
            messages.append({"role": "assistant", "content": response.content})

            if not calls:
                print("stopped: no tool call, stop_reason=%s" % response.stop_reason)
                break

            results = []
            for call in calls:
                print("[%d] do  %s" % (step, call.input))
                out = execute(page, call.input)
                payload = ([{"type": "image",
                             "source": {"type": "base64", "media_type": "image/png",
                                        "data": out["image"]}}]
                           if "image" in out
                           else [{"type": "text", "text": out["text"]}])
                results.append({"type": "tool_result",
                                "tool_use_id": call.id,
                                "content": payload})

            messages.append({"role": "user", "content": results})
            prune_images(messages, KEEP_SCREENSHOTS)
        else:
            print("stopped: hit MAX_STEPS=%d" % MAX_STEPS)

        print("done. url=%s title=%s" % (page.url, page.title()))
        browser.close()


if __name__ == "__main__":
    main()

Three details in there do real work. device_scale_factor=1 keeps one CSS pixel equal to one image pixel, so the model's coordinates match the viewport. prune_images replaces all but the newest three screenshots with a stub, because every turn otherwise re-sends every frame. And the for ... else fires only when the loop exhausts without a break, so the log tells you "finished" apart from "ran out of steps".

The final else branch in execute is deliberate too. The computer_20250124 tool version can emit actions this file does not implement — left_mouse_down and left_mouse_up for drags, hold_key, left_click_drag — and the honest response to an unimplemented action is a text tool result saying so, not a crash. The model reads that string and routes around it on the next turn, usually by picking a different approach to the same subgoal. Returning a refusal the model can see is the difference between a loop that recovers and a traceback that ends the run, and the same pattern applies to actions you decline on policy grounds rather than capability grounds.

Run it

1. 1
   Create a clean directory and virtualenv: mkdir cua && cd cua && python3 -m venv .venv && source .venv/bin/activate. Your prompt should now be prefixed with (.venv).
2. 2
   Install the two dependencies: pip install anthropic playwright. The last line should read Successfully installed anthropic-... playwright-....
3. 3
   Download the browser binary: playwright install chromium. You should see Downloading Chromium ... followed by a percentage counter, then Chromium ... downloaded to /root/.cache/ms-playwright/chromium-... (the path varies by OS).
4. 4
   Export your key from console.anthropic.com/settings/keys: export ANTHROPIC_API_KEY=sk-ant-.... Confirm with echo ${ANTHROPIC_API_KEY:0:7}, which should print sk-ant- and nothing else.
5. 5
   Save the two files above as page.html and agent.py, then check they are both there: ls prints agent.py page.html.
6. 6
   Run the loop: python agent.py. Each turn prints one usage line and one do line, and the last line prints the final URL.
7. 7
   Verify the loop actually closed. The final line should contain url=https://example.com/?email=ada%40example.com and title=Example Domain. If the URL is still file://.../page.html, the agent never submitted the form.

Output looks roughly like the block below. Coordinates, token counts and the exact action sequence differ on every run — the model may press Return instead of clicking Subscribe, and may take one screenshot you did not ask for. The shape is stable. The path is not.

[1] usage in=2714 out=131
[1] do  {'action': 'left_click', 'coordinate': [228, 161]}
[2] usage in=3901 out=96
[2] do  {'action': 'type', 'text': 'ada@example.com'}
[3] usage in=5089 out=118
[3] say I can see the email is filled in. Now clicking Subscribe.
[3] do  {'action': 'left_click', 'coordinate': [78, 215]}
[4] usage in=6207 out=74
[4] say The main heading reads: Example Domain
stopped: no tool call, stop_reason=end_turn
done. url=https://example.com/?email=ada%40example.com title=Example Domain

When it doesn't work

That last row is the one to sit with. The abort is the feature. A loop that screenshots, re-decides, and acts with no external bound is the common way these systems burn a budget, and no amount of prompt wording reliably prevents it — the stop has to live in code the model cannot write.

What one step of this costs

At 1024×768, one screenshot costs about 1,049 input tokens. That is arithmetic from Anthropic's documented image token estimate, (width px × height px) / 750: 786,432 / 750. With KEEP_SCREENSHOTS = 3, the image portion of the context tops out near 3,147 tokens no matter how long the task runs. Claude Sonnet 4.5 is listed at $3 per million input tokens and $15 per million output, so those three frames are roughly $0.0094 of input per request. Read your own usage lines for the real figure.

Pruning and prompt caching pull in opposite directions, which is worth knowing before you reach for both. Caching bills a cheaper rate for a prefix that is byte-identical to the previous request, so it rewards an append-only transcript. prune_images edits turns that are already in the transcript, which changes that prefix and invalidates the cache from the edit point onward. For a twelve-step browser task the pruning wins outright, because an unpruned transcript grows by a full frame every single turn. For longer runs, the trade has to be measured rather than assumed.

Output is tiny — an action JSON is a few dozen tokens. The cost of a computer-use agent is almost entirely the pictures, re-sent every turn. So the two levers that matter are how many screenshots you keep in context and how many turns the task needs, not which model you pick. If you are putting a price on something built this way, build the margin meter first.

How this differs from the vendor stacks

What you just built swaps the actuator. The loop is identical to the one in Anthropic's reference container; that container drives a real X11 desktop with xdotool inside Docker, so it can reach any application, not just a browser tab. OpenAI's version runs the same cycle through the Responses API: a computer_use_preview tool returns computer_call items, you execute them and reply with computer_call_output containing the next screenshot. Its one structural difference worth knowing is pending_safety_checks — items you must explicitly acknowledge before the loop continues.

DOM automation is in that table on purpose. If the page has stable selectors, or the service has an API, pixels are the wrong interface — they cost tokens, they break on redesigns, and they put a probabilistic step between you and a deterministic one. On OSWorld, a benchmark of real tasks on real operating systems, humans completed 72.36% while the best model in the paper managed 12.24%. That gap is why I treat screen control as a last resort rather than a default.

Before it touches anything that matters

Four changes, in the order I would make them.

Restrict navigation to an allowlist of domains and refuse anything else inside execute. A page can instruct the model, and the model will read it. Log every action to disk alongside the screenshot that produced it, so a wrong click is reconstructable afterwards. Keep the step cap outside the model permanently, and tie it to a wall-clock timeout as well as a count.

And do not hand it spend authority. I would not give a screenshot loop a saved payment method, and the reason is in the table above: it fails silently when the layout moves, and a silent failure on a checkout page is a purchase.

Start with the narrowest task that is genuinely annoying and has no API, and keep it bounded. When an API appears for that task, delete the agent.