Nova

A Computer Use implementation in Rust. Nova is a Model Context Protocol server that gives an LLM agent AX-first control of macOS and Windows: semantic UI reads/actions, screenshots, mouse, keyboard, scrolling, window/app introspection, OCR, and the clipboard — the "computer use" capability, built natively in Rust rather than wrapping a Python/JS automation stack.

Built directly on native platform APIs (macOS Accessibility/ScreenCaptureKit/ CoreGraphics and Windows UI Automation/Win32) — a single self-contained binary, no runtime to install. Connect it to any MCP client (Claude Desktop, an agent runtime, your own) over stdio or Streamable HTTP.

Tools

Related MCP server: macos-control-mcp

Requirements

• macOS 15+ (a transitive dependency, apple-metal, needs the macOS 15 SDK / Xcode 16+).

• Screen Recording permission — for screenshot / ocr / list_windows.

• Accessibility permission — for ax_read, semantic activation, and input.

Both are granted to the nova binary itself, not the app that launches it — see Permissions & code signing. On a dev build you must also sign nova, or the grant breaks on every rebuild.

Run

cargo run                      # stdio transport (default)
cargo run -- --http            # Streamable HTTP on 127.0.0.1:3100
cargo run -- --http --addr 0.0.0.0:8080

The Swift runtime that ScreenCaptureKit links is located via an LC_RPATH baked in by build.rs, so no DYLD_* environment variable is needed for cargo run/cargo test or the standalone binary.

Install

macOS 14+ (Apple Silicon or Intel — the release is a universal binary). Pick one:

# Homebrew (recommended) — puts `nova` on your PATH
brew install bigduu/tap/nova

# npx — no install; great for MCP configs
npx -y @bigduu/nova --version

# Prebuilt tarball — download, unquarantine, drop on PATH
curl -fsSLO https://github.com/bigduu/Nova/releases/latest/download/nova-vX.Y.Z-universal-apple-darwin.tar.gz
tar xzf nova-*.tar.gz
xattr -dr com.apple.quarantine ./nova   # only needed for the raw download
sudo mv nova /usr/local/bin/

# From source
cargo build --release      # produces target/release/nova

The release binary is ad-hoc signed, not notarized. Homebrew and npm install it without a Gatekeeper prompt; only the raw tarball download needs the one-time xattr -dr com.apple.quarantine above.

Use it from an MCP client

Claude Desktop (or any stdio MCP client) — add Nova to the client's MCP config. For Claude Desktop that's ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "nova": { "command": "npx", "args": ["-y", "@bigduu/nova"] }
  }
}

With brew install (or a binary on PATH) the command is simply "nova"; from source, use the absolute path to target/release/nova.

Restart the client; the Nova tools then appear to the agent. Grant Screen Recording and Accessibility — see Permissions & code signing (as a subprocess, the responsible process is usually the host app or your terminal, so grant that, not only the binary).

HTTP clients — run Nova as a server and connect over Streamable HTTP:

nova --http                       # 127.0.0.1:3100/mcp
nova --http --addr 0.0.0.0:8080   # reachable on the LAN

First calls. Call ax_read (optionally ax_read(window="<name>", mode="all")) for semantic content and controls, then ax_activate(snapshot_id, node_id) on an exact actionable node. Re-run ax_read after the action to verify semantic state. If AX/UIA coverage is absent or partial, use focused-window ocr for rendered text; use screenshot(window=...) / zoom_region only when pixels are necessary (layout, icon, color, image, canvas, or visual verification). All pointer tools use the pixel space of the most recent screenshot.

Permissions & code signing (macOS)

Two non-obvious things about how macOS grants Nova its Screen Recording and Accessibility permissions:

Nova is its own permission subject — grant the nova binary, not the launcher. As a standalone CLI (not inside an .app bundle), Nova does not inherit the Screen Recording grant of its parent process (your terminal, Claude Desktop, or a host app). Add the binary itself: System Settings → Privacy & Security → Screen Recording (and Accessibility) → + → ⌘⇧G → enter the path to target/release/nova, then enable it. A headless subprocess can't trigger the permission prompt, so adding it by path is the reliable way.

Sign it, or the grant breaks on every rebuild. cargo build produces an ad-hoc, linker-signed binary whose code-signing identity is a content hash (nova-<hash>) — it changes every build, so macOS treats each rebuild as a brand- new app and the grant stops applying. Sign with a stable self-signed identity so the grant persists:

cargo build --release
./scripts/dev-codesign.sh --release   # re-sign after EVERY build

The first run creates a Zenith Nova Code Signing identity in your login keychain (click Always Allow once if codesign prompts) and signs the binary with a fixed identifier (com.zenith.nova). Grant the two permissions once (above); later rebuilds, re-signed with the same cert, keep the grant.

Troubleshooting — screenshot fails with a "wedged" / "busy" capture error. All captures (and window enumeration) run in ONE shared per-user daemon (nova --capture-daemon, flock-elected, socket /tmp/nova-capture-<uid>-<hash>.sock), because replayd keys clients by executable path — two same-binary ScreenCaptureKit clients evict each other's XPC identity and wedge every new stream start. The daemon kills itself if a capture exceeds its 8s watchdog, and the client auto-recovers: kill+respawn the daemon, then (second failure) SIGKILL all nova capture processes and killall -9 replayd — wedges self-heal without manual action. If they don't: nova --selftest (probes ScreenCaptureKit in a sacrificial subprocess, then the daemon path) and read /tmp/nova-capture-worker.log (step trace) + /tmp/nova-capture-daemon.log (daemon stderr). Manual remedy = kill the processes holding streams (pkill -f -- --capture-daemon), NOT replayd: plain killall replayd is a no-op (replayd ignores SIGTERM), and even killall -9 replayd doesn't cure a wedge while a stream-holding client survives — it just reconnects and re-wedges the fresh replayd.

Coordinate grounding

A general LLM judging pixel coordinates off a downscaled screenshot is the main source of mis-clicks — so the primary path avoids pixels entirely.

• ax_read first (no image) — returns actionable controls and non-actionable readable content in deterministic tree order. A successful macOS read requires Accessibility but does not contact ScreenCaptureKit. permission_denied means fix that grant; it is not an instruction to take a screenshot.

• Fresh semantic action — call ax_activate with the returned snapshot and node IDs. Native AX/UIA and the browser DOM bridge are tried before a freshly revalidated element-center click. Stale generations fail closed; every activation attempt consumes its generation before provider dispatch, so read again after any result.

• OCR second — when coverage is absent/partial and the missing information is rendered text, use focused-window OCR and its returned text center with left_click(..., source="ocr_center").

• Screenshot/zoom last — use pixels for visual-only state or a surface with no semantic/text representation; coordinate clicks report route=visual_coordinate. Screenshot marks and click_mark remain available for compatibility.

When a screenshot is needed, all click/move/scroll tools work in the pixel space of the last screenshot — the server remembers that frame and maps clicks back to the real screen, so the model just "clicks what it sees":

• screenshot(window: "<name>") — capture a single window (substring of its title or app name) instead of the whole display. Smaller, sharper image → less context and far less downscaling → better precision. Later clicks map into that window.

• zoom_region(x, y, w, h) — magnify a rectangle of the last screenshot at native resolution (capturing only that rectangle). For reading small targets on surfaces that expose no Accessibility tree (canvas, games, custom views), where coordinates are the only option. A labeled coordinate grid is overlaid so the model reads positions straight off the axes.

Testing

The suite is split into fast, hermetic tests (run by default) and side-effecting end-to-end tests (opt-in, #[ignore]d).

Default — unit + hermetic integration tests

cargo test

Runs everything that has no side effects and needs no special permission:

• unit tests for coordinate scaling, the key/char keystroke maps, combo parsing, batch (de)serialization, and MCP tool registration;

• tests/e2e_interaction.rs — screenshot→logical coordinate mapping (via CGDisplay, no permission needed) and a non-destructive clipboard round-trip (snapshots and restores the clipboard).

This is what CI runs (see .github/workflows/ci.yml, macOS runner).

End-to-end tests (#[ignore]d)

These either post real input events (they move the cursor, click, scroll, or type into the focused window) or require Screen Recording permission, so they are excluded from cargo test and must be opted into. Run them on a desktop session where that's acceptable:

# all of them
cargo test -- --include-ignored

# or a single one
cargo test --test e2e_input mouse_move_roundtrips_through_cursor_position -- --ignored

mouse_move_roundtrips… is the strongest input check: it proves events actually reach the window server and that the screenshot→logical coordinate conversion is correct, all click paths share the same posting mechanism.

Run e2e_capture_worker single-threaded (-- --ignored --test-threads=1): the tests share one daemon/socket.

list_applications_returns_app_bundles (in e2e_input) is not ignored — it only reads Spotlight and is tolerant of a Spotlight-less CI host.

Lint & format

cargo fmt --all -- --check
cargo clippy --all-targets

Releasing (maintainers)

A version tag drives everything via .github/workflows/release.yml: it builds the universal binary (arm64 + x86_64, ad-hoc signed), smoke-tests it, and attaches it to a GitHub Release. When their secrets are present it also updates the Homebrew tap and publishes the npm wrapper; without them those steps skip, so the GitHub Release still ships.

# bump the version, then tag it (the tag must match Cargo.toml)
git tag v0.1.0
git push origin v0.1.0

One-time setup for the optional channels:

• Homebrew — gh repo create bigduu/homebrew-tap --public, then add a fine-grained PAT with Contents: write on that repo as the nova-repo secret HOMEBREW_TAP_TOKEN. (packaging/homebrew/nova.rb is the formula template.)

• npm — publish under the @bigduu scope; add an npm automation token as the secret NPM_TOKEN. The package (npm/) downloads the matching release binary on install.

License

MIT © bigduu