01What computer use is
Computer use lets an agent see a virtual machine's screen and work it — clicking, typing, and scrolling through the interface — not just run commands.
Think of it as the visual side of the VM. Commands are for scripting and builds. Computer use is for anything with a screen. Use it when the agent needs to:
- Click through an app it just built and confirm the interface really works.
- Drive a tool's window — the parts you can't reach from the command line.
- Check how something looks — a layout, a web page, a window — with a real screenshot.
Nucleic offers computer use in two forms, and they stack: semantic understanding builds on kernel-level use — it doesn't replace it. Both work with macOS and Linux virtual machines.
02Kernel-level use
Kernel-level use is the foundation. The agent works the machine from the outside, just as you would at the keyboard: it takes a screenshot, then moves the pointer, clicks, scrolls, and types.
With it, an agent can:
- Take a screenshot of the screen.
- Move and click the pointer, including press-and-drag.
- Scroll and type text, with modifier keys for shortcuts.
- Launch apps in the machine.
Together, these form a simple screenshot → decide → act loop: capture the screen, choose an action, perform it, then capture again to see what changed. It all happens quietly in the background — no window opens on your Mac, and your own mouse and keyboard are never captured. And there's nothing to set up. It's ready as soon as a base image is built.
03Semantic understanding
Semantic understanding builds on kernel-level use. It gives the agent a structured view of the interface — the on-screen controls as named elements, not just pixels. With it, the agent can:
- Act on a control by identity — press a specific button by its name — instead of guessing at coordinates.
- Read the interface's structure — which controls are present, their values, and what's focused — when it needs a precise picture of the screen.
This makes GUI automation far more reliable: the agent targets controls by what they are, not where they happen to sit. The trade-off? Reading and reasoning over the interface's structure increases token usage. Turn it on when you specifically want robust, by-identity control.
04Which to use
| Kind | What you get |
|---|---|
| Kernel-level use | See and drive the screen with screenshots, clicks, and typing. Runs in the background, with no setup. The right default for most GUI work. |
| Semantic understanding | Adds a structured, by-identity view of the interface for more reliable automation — at the cost of higher token usage. Builds on kernel-level use; it doesn't replace it. |
05Turning them on
Go to Settings ▸ Virtual Machines. Under Computer use, turn on Kernel-level use. Once it's on, Semantic understanding appears beneath it — turn that on too if you want the structured view. Both settings apply to whichever virtual machines you've enabled.
Computer use needs a built virtual machine, so you'll need a base image first. See Virtual machines: overview.