Computer Use

What computer-use agents can do, and how to run them safely

How AI agents see the screen, click and type, which jobs they're good at today, and the guardrails to put in place first.

Beginner 8 min read 5 sections 2026-10-31

A computer-use agent is an AI model that operates a computer the way you do. It looks at the screen, decides where to click or what to type, and repeats until the task is done. Because it works through the same interface you use, it can handle apps and websites that have no API at all.

How it works

Each step is a loop:

  1. The agent looks at the current state of the screen or the web page.
  2. The model decides on the next action, such as clicking a button, typing into a field or scrolling.
  3. The action runs, and the agent looks again to see what changed.

The loop keeps going until the agent decides the task is finished, gets stuck, or reaches a point where it should ask you.

Two ways an agent can see

Agents read the screen in one of two ways, and the difference shows up in speed and reliability.

  • Screenshots. The model gets a picture of the screen and works out where to click from pixels. This works with any app, including desktop software, but it needs a model that's good at reading images, and a busy screen can trip it up.
  • The page's structure. For web pages, a tool like Playwright MCP gives the model the page's accessibility tree: a text list of the buttons, links and fields on the page, each with a label. The model picks an element by its reference instead of guessing coordinates, which is faster and more predictable. It only works inside a browser.

For anything in a browser, start with the structured approach. Use screenshot-based agents for desktop apps or pages where the structure is missing or misleading.

What it's good at today

  • Filling in forms with data from a spreadsheet or document
  • Clicking through admin dashboards to collect or change settings
  • Testing your own web app the way a user would
  • Repetitive research, like checking the same detail across many sites

It's slower than an API and can misread a page, so it fits jobs where a person would otherwise do the clicking. If a task has an API or a command-line tool, use that instead.

Guardrails before you start

Heads up

  1. Use a separate browser profile or a VM

  2. Stay in the loop on anything irreversible

  3. Watch for instructions on the page

  4. Start small and watch the first runs

Try it

The quickest way to see this for yourself is to give a coding agent a browser. Let Claude Code drive a browser with Playwright MCP walks through the setup in about ten minutes.

Keep going