Skip to content
Neural Daily

ToolsNews

agent-desktop: open computer use without screenshots

🇭🇺 Olvasd el magyarul →

Vercel Labs has released agent-desktop under Apache 2.0. The Rust CLI reads and operates native apps through the operating system's accessibility tree, so any shell-capable coding agent can drive desktop software without a vision model.

By: Mikola Máté Patrik · · 2 min read

GitHub page of the vercel-labs/agent-desktop repository
Image: Vercel · Screenshot · Screenshot of the official Vercel Labs GitHub repository · Source

What it is

agent-desktop is not an agent but a tool that agents call. It is a Rust CLI and C library with no embedded LLM and no GUI. Every command prints one JSON envelope and exits, and the observe, act and verify loop stays in your agent. Version 1.0.2 shipped on October 30 and runs on macOS 13+, Windows 10 1809+ and Ubuntu 24.04+, on x64 and ARM64, with no GPU.

How it differs

Model-side computer use tools from Anthropic and OpenAI work from screenshots and pixel coordinates. Anthropic's docs put one screenshot at roughly 1,000 to 1,800 input tokens. agent-desktop returns a structured snapshot instead: every actionable element gets a ref, and the agent acts on those refs.

  • Strict refs: when the UI changes, the agent gets STALE_REF and takes a fresh snapshot. There is no guessing, and resolution fails closed.
  • Skeleton mode: dense apps are read only 3 levels deep. The builder reports 78% to 96% fewer tokens in Slack, VS Code and Notion, and 273 instead of 3,811 tokens per look in Apple Maps.
  • Runs in the background: ref actions do not move the pointer or take focus, so you can keep using the machine. Multiple agents can work in parallel.

These are the builder's own figures, and there is no independent computer use benchmark yet (MarkTechPost).

Who it's for

Anyone already working with Claude Code, Codex or another CLI agent who wants to automate native apps too. Install with npm install -g agent-desktop. macOS needs Accessibility permission, Windows does not by default. Agents learn the tool from bundled skill docs. The approach is sensible: if the OS already knows what is on screen, there is no need to infer it from pixels.

    Produced with the help of AI, reviewed by an editor. How we make our articles

    Reviewed and edited by the Neural Daily desk. About

    Related articles