Computer use for macOS VMs

01What computer use is

Computer use lets an agent see and drive a macOS VM's GUI through the mac_vm_computer tool. Think of the VM as a Mac dev simulator the agent can look at and operate.

It's the visual counterpart to mac_vm_exec, which runs headless shell commands inside the VM. Where mac_vm_exec is for scripting and builds, computer use is for anything with a screen. Use it when the agent needs to:

• Look at — or click through — an app it just built, to confirm the UI behaves as it should.
• Drive Xcode or the Simulator UI — the parts you can't reach from the command line.
• Check how something renders — a layout, a web page, a window — with a real screenshot of the VM's display.

02Why it needs no setup

In the default path, screen capture and input both happen host-side. Screenshots come from the VM's display, and clicks and keystrokes are delivered to it from the outside. The agent never reaches into the guest.

That means there's nothing to install and nothing to configure inside the VM:

• Nothing runs inside the guest. No helper app, no login agent — capture and input happen on your Mac.
• The guest grants no permissions. No Accessibility, no Screen Recording — because no in-guest software is asking for them.
• The guest's security posture stays untouched. Nothing inside the guest changes to capture the screen or inject input.

Because nothing is needed inside the guest, the default path works with any installed base image — there's no special build variant to create first.

03Turning it on

In Settings ▸ Virtual Machines, under Computer use, turn on Kernel-level use. Sessions then expose the mac_vm_computer tool to the agent.

The first call to the tool boots that session's VM — that takes a minute or two. Later calls reuse the running VM, so they're fast. You never launch anything by hand: the agent invokes the tool, and the session's VM spins up on demand.

Two things need to be in place first:

• The macOS VM service is on. See macOS virtual machines: overview for requirements and how to enable it.
• A base image is built. Computer use runs against an installed VM, so you need a built base first — see Building the base image.

04What the agent can do

Through the mac_vm_computer tool, an agent can:

• Take a screenshot of the VM's screen.
• Move and click the mouse, including press-and-drag.
• Scroll with a real scroll wheel.
• Type text, with modifier keys for shortcuts.
• Launch apps in the guest.

These add up to a simple screenshot → decide → act loop: the agent captures the screen, chooses an action, performs it, then captures the screen again to see what changed — just as a person watches a display and reacts.

05When to add the semantic agent

The default pixel path handles most GUI work. But you can layer an optional in-guest semantic AX agent on top. Add it when you want to:

• Act on a UI element by identity — press a specific button by its role and title — instead of guessing and clicking pixel coordinates.
• Inspect an app's accessibility tree — roles, values, and focus state — when debugging a Mac or iOS app's UI.

The trade-off between the two paths is setup versus capability: