PyMCPAutoGUI 🖱️⌨️🖼️ - GUI Automation via MCP

Supercharge your AI Agent's capabilities! ✨ PyMCPAutoGUI provides a bridge between your AI agents (like those in Cursor or other MCP-compatible environments) and your computer's graphical user interface (GUI). It allows your agent to see the screen 👁️, control the mouse 🖱️ and keyboard ⌨️, and interact with windows 🪟, just like a human user!

Stop tedious manual GUI tasks and let your AI do the heavy lifting 💪. Perfect for automating repetitive actions, testing GUIs, or building powerful AI assistants 🤖.

🤔 Why Choose PyMCPAutoGUI?

• 🤖 Empower Your Agents: Give your AI agents the power to interact directly with desktop applications.

• ✅ Simple Integration: Works seamlessly with MCP-compatible clients like the Cursor editor. It's plug and play!

• 🚀 Easy to Use: Get started with a simple server command. Seriously, it's that easy.

• 🖱️⌨️ Comprehensive Control: Offers a wide range of GUI automation functions from the battle-tested PyAutoGUI and PyGetWindow.

• 🖼️ Screen Perception: Includes tools for taking screenshots and locating images on the screen – let your agent see!

• 🪟 Window Management: Control window position, size, state (minimize, maximize), and more. Tidy up that desktop!

• 💬 User Interaction: Display alert, confirmation, and prompt boxes to communicate with the user.

Related MCP server: helix-pilot

🛠️ Supported Environments

• Operating Systems: Windows, macOS, Linux (Requires appropriate dependencies for pyautogui on each OS)

• Python: 3.11+ 🐍

• MCP Clients: Cursor Editor, any client supporting the Model Context Protocol (MCP)

🚀 Getting Started - It's Super Easy!

1. Installation (Recommended: Use a Virtual Environment!)

Using a virtual environment keeps your project dependencies tidy.

# Create and activate a virtual environment (example using venv)
python -m venv .venv
# Windows PowerShell
.venv\Scripts\Activate.ps1
# macOS / Linux bash
source .venv/bin/activate

# Install using pip (from PyPI or local source)
# Make sure your virtual environment is active!
pip install pymcpautogui # Or pip install . if installing from local source

(Note: pyautogui might have system dependencies like scrot on Linux for screenshots. Please check the pyautogui documentation for OS-specific installation requirements.)

2. Running the MCP Server

Once installed, simply run the server from your terminal:

# Make sure your virtual environment is activated!
python -m pymcpautogui.server

The server will start and listen for connections (defaulting to port 6789). Look for this output:

INFO:     Started server process [XXXXX]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:6789 (Press CTRL+C to quit)

Keep this terminal running while you need the GUI automation magic! ✨

✨ Seamless Integration with Cursor Editor

Connect PyMCPAutoGUI to Cursor (@ symbol) for GUI automation directly within your coding workflow.

1. Open MCP Configuration: In Cursor, use the Command Palette (Ctrl+Shift+P or Cmd+Shift+P) and find "MCP: Open mcp.json configuration file".

2. Add PyMCPAutoGUI Config: Add or merge this configuration into your mcp.json. Adjust paths if needed (especially if Cursor isn't running from the project root).

   {
       "mcpServers": {
           // ... other MCP server configs if any ...
           "PyMCPAutoGUI": {
               // Sets the working directory. ${workspaceFolder} is usually correct.
               "cwd": "${workspaceFolder}",
   
               // Command to run Python. 'python' works if the venv is active in the terminal
               // where Cursor was launched, or specify the full path.
               "command": "python", // Or ".venv/Scripts/python.exe" (Win) or ".venv/bin/python" (Mac/Linux)
   
               // Arguments to start the server module.
               "args": ["-m", "pymcpautogui.server"]
           }
           // ... other MCP server configs if any ...
       }
   }

   (Tip: If mcp.json already exists, just add the "PyMCPAutoGUI": { ... } part inside the mcpServers object.)

3. Save mcp.json. Cursor will detect the server.

4. Automate! Use @PyMCPAutoGUI in Cursor chats:

   Example: @PyMCPAutoGUI move_to(x=100, y=200) @PyMCPAutoGUI write(text='Automating with AI! 🎉', interval=0.1) @PyMCPAutoGUI screenshot(filename='current_screen.png') @PyMCPAutoGUI activate_window(title='Notepad')

🧰 Available Tools

PyMCPAutoGUI exposes most functions from pyautogui and pygetwindow. Examples include:

• Mouse 🖱️: move_to, click, move_rel, drag_to, drag_rel, scroll, mouse_down, mouse_up, get_position

• Keyboard ⌨️: write, press, key_down, key_up, hotkey

• Screenshots 🖼️: screenshot, locate_on_screen, locate_center_on_screen

• Windows 🪟: get_all_titles, get_windows_with_title, get_active_window, activate_window, minimize_window, maximize_window, restore_window, move_window, resize_window, close_window

• Dialogs 💬: alert, confirm, prompt, password

• Config ⚙️: set_pause, set_failsafe

For the full list and details, check the pymcpautogui/server.py file or use @PyMCPAutoGUI list_tools in your MCP client.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details. Happy Automating! 😄