browsercontrol

The idea

Most browser tools hand a model a DOM tree and hope it writes a working selector. BrowserControl hands it a picture.

Every action returns a fresh screenshot with numbered red boxes drawn over the interactive elements, plus the same list as text. The agent acts by number:

click(5)
type_text(3, "hello world")
upload_file(7, "/path/to/resume.pdf")

No selectors to guess, none to break on the next redesign. This is the Set of Marks pattern, used as the primary interface rather than a fallback.

Related MCP server: Helm

Quick start

uv add browsercontrol      # or: pip install browsercontrol

Chromium installs itself on first run. If that fails, run python -m playwright install chromium once.

Point your client at the browsercontrol command:

{
  "mcpServers": {
    "browsercontrol": {
      "command": "browsercontrol"
    }
  }
}

No install at all, if you prefer: "command": "uvx", "args": ["browsercontrol"].

Restart the client and ask it for something:

"Open Hacker News and tell me the top story."

Full walkthrough: Getting started.

The one rule

Element numbers expire after every action. The map is rebuilt from scratch on every click, type, scroll, navigation, and screenshot — element 7 before a click is rarely element 7 after it.

So an agent should never plan two clicks from one screenshot. It acts, reads the new screenshot, finds the target again, then acts again. Almost every failure worth debugging traces back to this.

Two corollaries:

• Only what's in the viewport gets marked. No number usually means "below the fold" — scroll, then look again.

• Cross-origin iframes are never marked. Open shadow roots and same-origin frames are traversed and numbered; third-party payment fields and embedded widgets can't be. That's a boundary of the browser, not a bug.

Tools

39 tools across seven categories. Everything that touches the page returns an annotated screenshot plus the element map.

A few behaviours that aren't obvious from the names:

• type_text uses Playwright's fill() — it replaces the field, it doesn't append.

• click resolves the topmost element at the mark's centre, so a cookie banner overlapping your target will eat the click. Dismiss overlays first.

• upload_file goes through set_input_files, which works on sites where driving the file picker doesn't.

• screenshot(annotate=True, full_page=True) returns a clean image — full-page captures can't be marked.

• get_page_content returns Markdown, capped at 30 KB.

• Recordings are Playwright traces. Open one with npx playwright show-trace ~/.browsercontrol/recordings/<name>.zip.

Teach your agent to use it well

Connecting the server tells an agent which tools exist. The bundled agent skill tells it how to use them well — that numbers expire, that a banner will swallow a click, that a missing number means scroll.

mkdir -p ~/.claude/skills/browsercontrol
curl -fsSL https://raw.githubusercontent.com/adityasasidhar/browsercontrol/main/skills/browsercontrol/SKILL.md \
  -o ~/.claude/skills/browsercontrol/SKILL.md

It loads only when a task actually involves a browser, so it costs one line of context the rest of the time. Agents that would rather read the docs directly can start at llms.txt.

Configuration

Environment variables, read once at startup.

BROWSER_HEADLESS=false BROWSER_VIEWPORT_WIDTH=390 BROWSER_VIEWPORT_HEIGHT=844 browsercontrol

Why this one

The session is a real Chromium profile (launch_persistent_context), so cookies, localStorage and logins survive restarts — log in once and your agent stays logged in. DevTools are first-class: console, network timing, uncaught exceptions, performance and computed styles are tools, not an afterthought. And it runs entirely on your machine — no LLM API key, no cloud round-trip, no per-action cost, nothing leaving the box.

Honest positioning: if you're happy driving the accessibility tree, Playwright MCP is excellent. If you want an LLM planning the interactions for you, look at Stagehand or Browser-Use. BrowserControl's niche is vision-first, fully local browser control with devtools built in. Longer comparison in the docs.

Development

git clone https://github.com/adityasasidhar/browsercontrol
cd browsercontrol
uv sync
uv run playwright install chromium --with-deps

uv run pytest                                  # tests (Playwright is mocked — no real browser)
uv run ruff check . && uv run ruff format .    # lint + format
uv run fastmcp dev browsercontrol/server.py    # dev server with the MCP inspector

browsercontrol/
├── server.py     # FastMCP instance, lifespan, tool registration
├── browser.py    # BrowserManager: Playwright lifecycle, element detection, SoM rendering
├── config.py     # env-var config
└── tools/        # one module per category, each exporting register_*_tools(mcp)

CI runs the suite on Ubuntu, Windows and macOS with ruff, mypy --strict, bandit and pytest.

Contributing

Issues and PRs welcome — see CONTRIBUTING.md. New tools should come with a test for the happy path and the error path; the existing tests/test_*.py files show the pattern.

On the list, if you're looking for somewhere to start: Firefox and WebKit support, mobile device presets, network mocking, DOM diffing between snapshots, and accessibility auditing.

License

MIT. Built on FastMCP and Playwright.