Computer and browser use

Computer use lets the model operate a real screen: it looks at a screenshot, picks one action (a click, a keystroke, a scroll) your code carries it out, and a new screenshot goes back. Anthropic names the mechanism precisely. “The repetition of steps 3 and 4 without user input is referred to as the ‘agent loop’”[1]: those two steps are Claude responding with a tool use request, and your application responding with the results of evaluating it. OpenAI’s own tool repeats the same exchange “until the model stops returning computer_call items”[2], and Google documents your application capturing a new screenshot and sending it back “to request the next step”[3].

One action, requested from one screenshot, is level 4: the model picks it, your code runs it. But that is not how any maker ships computer use: every real use is the loop itself, exited by the model, not by your code, which is single agent behavior, level 5. This page shows one action, on purpose, because that is as far as level 4 goes; treat every screenshot after the first as already the next level up.

This page provides primary references and illustrative examples. The examples are scripted, not measured model runs; source references do not establish the correctness of every implementation or outcome.

[search-box] field "Search" value=""
[search-button] button "Search"
[cookie-accept] button "Accept all cookies"
[delete-account] link "Delete my account"

# The only elements this run is permitted to act on, regardless of what the model asks for.
# "cookie-accept" and "delete-account" exist on the screen and can be clicked in the sense that a
# tool call naming them is well-formed -- but they are not on this list, so the code refuses them
# before anything happens. This is the allowlist the real makers describe: a fixed set of safe
# actions, not a judgment call made per request.
ALLOWED_ELEMENT_IDS = frozenset({"search-box", "search-button"})

SCREEN = [
    {"id": "search-box", "kind": "field", "label": "Search", "value": ""},
    {"id": "search-button", "kind": "button", "label": "Search"},
    {"id": "cookie-accept", "kind": "button", "label": "Accept all cookies"},
    {"id": "delete-account", "kind": "link", "label": "Delete my account"},
]

def run(
    question: str,
    model: Model,
    embedder: Embedder | None,
    tracer: Tracer,
    *,
    screen: list[dict] = SCREEN,
    allowed_ids: frozenset[str] = ALLOWED_ELEMENT_IDS,
) -> Answer:
    del embedder  # this level acts on a screen, not a document index
    screen_text = render_screen(screen)
    tracer.record(kind="code", decided_by="code", title="Render the screen as text", detail=f"{len(screen)} elements")

    messages = [
        Message(role="system", content=SYSTEM_PROMPT),
        Message(role="user", content=f"Screen:\n{screen_text}\n\nTask: {question}"),
    ]
    completion = model.complete(messages, tools=TOOLS, max_tokens=200)

    if not completion.tool_calls:
        tracer.record(
            kind="model",
            decided_by="model",
            title="Model takes no action",
            detail=completion.text[:200],
            tokens_in=completion.tokens_in,
            tokens_out=completion.tokens_out,
            ms=completion.ms,
        )
        return Answer(text=completion.text or "No action taken.", citations=[])

    call = completion.tool_calls[0]
    element_id = str(call.arguments.get("id", ""))
    tracer.record(
        kind="model",
        decided_by="model",
        title=f"Model chooses {call.name}({element_id})",
        detail=str(call.arguments),
        tokens_in=completion.tokens_in,
        tokens_out=completion.tokens_out,
        ms=completion.ms,
    )

    if call.name not in {"click", "type"} or element_id not in allowed_ids:
        tracer.record(
            kind="code",
            decided_by="code",
            title="Action refused: not on the allowlist",
            detail=f"{call.name}({element_id!r})",
        )
        return Answer(text=f"Refused: {call.name}({element_id!r}) is not on the allowlist for this run.", citations=[])

python -m examples.computer_use --model stub:scripted

Failure modes

Cost and latency

Illustrative, not measured: no result file exists for this technique yet, so every number below is a worked example. See docs/EVALS.md for how a real run is recorded and scored.

How to Evaluate It

This example does not answer a question about the document corpus, so the site’s 60-question set has nothing to grade it against. It is registered with the runner as not scored, with that reason (see docs/EVALS.md): asking scripts/eval_run.py for it by name prints the reason and stops. What a real computer-use run is measured on instead: task success (did the sequence of actions reach the stated goal), the refusal rate on actions outside a stated allowlist, and how often a run stops at a confirmation point rather than completing an irreversible action on its own. None of those are one-shot numbers this level-4 slice can produce; they only mean something over the level-5 loop a real run actually is.

Run it

Try it

1. Use it

   Find an agent product that can operate a browser or a desktop. Ask it to do something with a real consequence, like sending a message: does it stop to confirm first, or just do it?

2. Build it

   Run python -m examples.computer_use --model stub:scripted from the repo root: the model types "warranty" into the search box and the code does it. Change that call in SCRIPTED (examples/computer_use/__main__.py) to click delete-account: the run prints Refused: the id is not on ALLOWED_ELEMENT_IDS. Add it there and the click runs. Only your code changed.

3. Either lane

   Take a failure mode above: how would you test for it in a product you use?