screen-mcp

Linux Wayland computer-use MCP for Claude Code and Grok — screenshot, click, type, scroll, drag, read any desktop app.

screen-mcp is an MCP server and plugin for Claude Code and Grok for desktop automation on GNOME/Wayland. It captures any monitor through PipeWire, drives pointer and keyboard via the xdg-desktop-portal RemoteDesktop portal, and optionally reads the screen with OCR plus OmniParser icon grounding. Pure Python, CPU-only — built for AI agents that need real computer-use on native Linux apps, not just a browser.

It ships the MCP tools plus a drive-screen skill that encodes the locate → ground → act → confirm/watch loop so the model uses screenshots and clicks well out of the box. screen_watch (default ~1 fps) is how agents see like humans: thrashing graphs, loaders, and canvases fail visual QA with a jitter verdict instead of slipping past a single freeze-frame. Developers get productivity tools that reach every visible window — including native Wayland apps that xdotool and XTEST cannot touch.

Install

Claude Code

/plugin marketplace add 88plug/claude-code-plugins
/plugin install screen-mcp@88plug

Grok Build

grok plugin marketplace add 88plug/claude-code-plugins
grok plugin install screen-mcp@88plug --trust

Then install system packages (once per machine) and Python deps (plugin venv). The marketplace cannot install these for you.

# --- system (pick your distro) ---
# Debian / Ubuntu
sudo apt install python3-gi python3-gi-cairo gir1.2-gstreamer-1.0 \
  gstreamer1.0-tools gstreamer1.0-plugins-base gstreamer1.0-plugins-good \
  gstreamer1.0-libav pipewire pipewire-pulse xdg-desktop-portal-gnome \
  wl-clipboard fonts-dejavu

# Arch / Manjaro
sudo pacman -S python-gobject gobject-introspection \
  gstreamer gst-plugins-base gst-plugins-good gst-libav \
  pipewire pipewire-pulse xdg-desktop-portal-gnome \
  wl-clipboard ttf-dejavu

# --- Python (required: numpy + Pillow; optional OCR later) ---
# From the installed plugin root (or a clone):
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
# minimum only:  .venv/bin/pip install 'numpy>=1.26' 'Pillow>=10.0'

Related MCP server: desk-mcp

Quickstart

On first use the desktop portal asks which monitor(s) to share. Pick one, then:

Take a screenshot of my desktop and tell me which window is focused.

You get a labeled capture plus the focused-window name within a few seconds. The portal restore token is cached at ~/.config/mcp-screen so later runs are silent.

Features

Principles — The Agent Oath

screen-mcp is a reference enforcer of The Agent Oath. The user-takeover guard yields the instant a human moves the mouse (STOPPED result) — §2 human agency and §11 human oversight, executable. Opt-in ack gate (§7) and on-screen visibility of every action (§5) round it out.

MCP tools

Every action also accepts space (view / desktop / norm, default view — coords as seen in the last screenshot), shot: true for a post-action shot, verify: true to warn on no-change misclicks, force: true to bypass the user-takeover guard, and element: <id> to click an element id from the last annotated shot.

Requirements

• Linux + Wayland + GNOME. The awareness layer uses a bundled GNOME Shell extension; AT-SPI is the fallback for GTK apps.

• Python 3.10+ runtime (tested on 3.14). CI pins 3.12 because optional the OCR backend is rapidocr (v3), which supports 3.8-3.x.

• GStreamer >= 1.28 (uses leaky-type; the older drop= was removed in 1.28).

• PipeWire and xdg-desktop-portal-gnome.

• wl-clipboard for the Unicode paste path in screen_type.

• A DejaVu Sans Bold font for Set-of-Marks labels (falls back to PIL's default).

Grounding is CPU-only by design: the server hard-disables the GPU (CUDA_VISIBLE_DEVICES="") for predictable latency and no driver flake.

Install the system deps before the Python deps. See requirements.txt for the full pacman / apt one-liners.

# Arch / Manjaro
sudo pacman -S python-gobject gobject-introspection \
               gstreamer gst-plugins-base gst-plugins-good gst-libav \
               pipewire pipewire-pulse xdg-desktop-portal-gnome \
               wl-clipboard ttf-dejavu

# Python deps
pip install -r requirements.txt

The bundled window-info extension gives the awareness layer reliable focused-window and window-list data, and lets screen_focus activate windows directly. Installing it needs a one-time Wayland re-login.

gnome-shell-extension/window-info@local/install.sh
gnome-extensions enable window-info@local

For the kernel input backend, grant access to /dev/uinput by adding your user to the input group. The launcher (bin/screen-mcp) fails with a clear message if a required dep is missing, so a misconfigured install never silently half-works.

Manual MCP setup

If you are not using the plugin, wire the server via the launcher (resolves host Python / .venv through scripts/run-python.sh — never bare python3 on thin Claude spawn PATH). Add to ~/.claude.json under mcpServers:

{
  "screen": {
    "command": "/path/to/screen-mcp/bin/screen-mcp"
  }
}

Configuration

screen-mcp reads optional environment variables. A few common ones:

• MCP_SCREEN_GUARD=1 — enable the ack gate. Destructive combos (Ctrl+W, Alt+F4), OCR-matched destructive keywords, and out-of-allowlist actions block unless the caller passes ack=<reason>.

• MCP_SCREEN_APPS="firefox,terminal" — with the guard on, restrict actions to this allowlist of focused apps.

• MCP_SCREEN_AMBIENT=0 — disable the ambient SENSE hint block.

• MCP_SCREEN_CPU_THREADS=6 — ONNX intra-op thread count for OmniParser.

• MCP_SCREEN_MAX_EDGE=2576 — screenshot downscale target (long edge).

Development

pytest -q          # tests run without a live D-Bus (conftest stubs)

Edit a .py, then call screen_reload in the running session to re-exec the server in place while preserving the MCP connection. On any tool exception the dispatcher writes the full traceback to /tmp/screen_err.txt; read it when debugging crashes.

Contributing

Issues and pull requests are welcome. Please run pytest -q before opening a PR. See CLAUDE.md for architecture notes and the hard-won ops details behind the capture and input layers.

License

FSL-1.1-ALv2 © 2026 88plug. The Functional Source License converts to Apache 2.0 two years after each release.