AIsistent
AIsistent is an MCP server for non-intrusive RDP and GUI automation — with zero footprint on the remote machine. It provides the following capabilities:
Capture Screen: Take a screenshot of the RDP window (macOS, via Quartz) or full primary monitor (Windows/Linux, via MSS).
Run OCR: Extract text from screenshots using Apple Vision on macOS (~0.05s, Neural Engine) or EasyOCR on Windows/Linux (CPU/CUDA).
Detect UI Buttons/Icons: Identify buttons and UI elements using YOLOv8, with automatic GPU backend selection: CUDA (NVIDIA), MPS (Apple Silicon), or CPU.
Inject Mouse Clicks: Simulate clicks at percentage-based coordinates (resolution-independent), via RDP protocol or pyautogui.
Manage RDP Connections: Connect to and disconnect from headless RDP sessions at runtime, supporting both GUI (local window) and headless (direct framebuffer) modes.
Benchmark Performance: Run OCR and YOLO inference benchmarks using a custom or synthetic image, returning timing and detection statistics as JSON.
Integrate with MCP Clients: Works with Hermes, OpenCode, Claude Desktop, Cursor, and other MCP-compatible clients.
Allows Hermes AI agent to perform screen capture, OCR, YOLO button detection, and click injection for desktop automation.
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@AIsistentOCR the current RDP screen"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
AIsistent
MCP server for non-intrusive RDP automation. OCR, YOLO button detection, and click injection — zero footprint on the remote machine.
GUI mode (default) | RDP headless mode | |
Capture | macOS RDP window / MSS fullscreen | Direct RDP framebuffer (simple-rdp) |
OCR | Apple Vision / EasyOCR | Apple Vision / EasyOCR |
Detection | YOLO (CUDA / MPS / CPU) | YOLO (CUDA / MPS / CPU) |
Click | pyautogui | RDP protocol input channel |
Quick Start
# macOS (Apple Silicon)
pip install aistent[apple]
# Windows / Linux (CPU)
pip install aistent[cpu]
# Windows (NVIDIA CUDA)
pip install aistent[cuda]
# RDP headless (any OS)
pip install aistent[rdp]
# Everything
pip install aistent[all]
aisistent # GUI mode (default)
aisistent --mode rdp --host HOST # RDP headless modeRelated MCP server: Computer Use MCP
Tools
Tool | Description |
| Open a headless RDP connection (switches transport to RDP mode) |
| Close the RDP connection and switch back to GUI mode |
| Capture screen via active transport (GUI or RDP) |
| OCR: Apple Vision (Mac) or EasyOCR (CPU/CUDA) |
| YOLOv8 button detection on CUDA / MPS / CPU |
| Click injection at percentage-based coordinates |
| Run performance benchmark on OCR + YOLO (returns JSON) |
Configuration
Env var | Default | Description |
|
| Path to YOLO weights file |
|
| Screenshot temp directory |
| — | RDP server hostname/IP (for headless mode) |
| — | RDP username |
| — | RDP password |
| — | RDP domain (optional) |
Cross-Platform Hardware Detection
Hardware is auto-detected at import time in aisistent/platform.py:
Backend | Detection | dtype | Use Case |
CUDA (NVIDIA) |
|
| Windows/Linux with NVIDIA GPU |
MPS (Apple) |
|
| macOS Apple Silicon (M1–M4) |
CPU | fallback |
| Any OS, no GPU |
Install GPU backends
# CUDA (NVIDIA)
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124
# MPS (Apple) — included in default torch on macOS
pip install torch torchvisionBenchmark
Run a quick performance test from the command line:
aisistent-bench # captures a real screenshot & benchmarks
aisistent-bench --synthetic # use synthetic image (no screen capture)
aisistent-bench --skip-ocr # YOLO only
aisistent-bench --image screenshot.png # use your own image
aisistent-bench --device cpu # force CPU backendOr via MCP tool call:
benchmark(image_base64: "") # empty = real screenshot, or pass base64Real-world performance (Apple MacBook M5 — MPS GPU)
Benchmark on a real 1920×1080 desktop screenshot with text, buttons, and UI elements:
Platform : macOS (Apple Silicon M5)
Device : MPS
──────────────────────────────────────
Capture : 0.22s
OCR : 0.43s — 102 texts detected
YOLO : 0.76s — 59 buttons detected
──────────────────────────────────────
Total : ~1.4sStep | Time | Throughput |
Screen capture | ~0.22s | — |
Apple Vision OCR (Neural Engine) | ~0.43s | ~237 texts/sec |
YOLOv8 inference (MPS float16) | ~0.76s | ~78 detections/sec |
End-to-end | ~1.4s | — |
These numbers are from the same machine running both the MCP server and the benchmark — no overhead from network or RDP. On NVIDIA CUDA, YOLO inference is typically 0.3–0.5s (RTX 3060+).
MCP Client Integration
AIsistent implements the standard MCP (Model Context Protocol), so it works with any MCP client. Below are detailed setup instructions for each platform.
Hermes MCP
Hermes is an AI agent that uses MCP tools to interact with your computer.
1. Install AIsistent
# macOS (Apple Silicon — Apple Vision OCR + MPS GPU)
pip install aistent[apple]
# Windows/Linux CPU
pip install aistent[cpu]
# Windows with NVIDIA GPU
pip install aistent[cuda]2. Locate Hermes config file
OS | Path |
macOS |
|
Windows |
|
Linux |
|
3. Add AIsistent to Hermes config
{
"mcpServers": {
"aisistent": {
"command": "aisistent",
"type": "stdio"
}
}
}If AIsistent is not on your PATH, use the full path:
{
"mcpServers": {
"aisistent": {
"command": "/path/to/venv/bin/aisistent",
"type": "stdio"
}
}
}4. Start Hermes
hermesHermes will auto-discover AIsistent's tools on startup. You should see:
👁️ AIsistent — capture_rdp_screen, run_apple_ocr, detect_rdp_buttons, inject_rdp_click, benchmarkExample: Hermes asks AIsistent to read the screen
> What's on my screen right now?Hermes will:
Call
capture_rdp_screen→ gets screenshotCall
run_apple_ocr(image)→ extracts all textCall
detect_rdp_buttons(image)→ finds buttonsReturns a structured summary of what's on screen
Example: Hermes clicks a button via AIsistent
> Open Chrome and go to youtube.comHermes will:
Call
capture_rdp_screen→ sees desktopCall
detect_rdp_buttons(image)→ finds Chrome icon coordinatesCall
inject_rdp_click(12.5, 8.3)→ clicks ChromeRepeats capture → detect → click until done
OpenCode
OpenCode is an agentic CLI that also supports MCP tools.
1. Install AIsistent
pip install aistent[all]2. Add to OpenCode config
Create or edit ~/.config/opencode/opencode.jsonc:
{
"mcpServers": {
"aisistent": {
"command": "aisistent",
"type": "stdio"
}
}
}Or per-project, add to .opencode.jsonc in your project root:
{
"mcpServers": {
"aisistent": {
"command": "aisistent",
"type": "stdio"
}
}
}3. Verify it works
opencodeThen ask:
capture the screen and tell me what applications are openOpenCode will call capture_rdp_screen → run_apple_ocr and return the result.
Claude Desktop
Claude Desktop supports MCP tools via its config file.
1. Locate Claude Desktop config
OS | Path |
macOS |
|
Windows |
|
2. Add AIsistent
{
"mcpServers": {
"aisistent": {
"command": "aisistent",
"type": "stdio"
}
}
}3. Restart Claude Desktop
Claude will show a hammer icon with AIsistent's available tools.
Cursor
Cursor IDE supports MCP tools.
1. Open Cursor settings
Settings → Features → MCP Servers
2. Add server
Name: AIsistent
Type: stdio
Command: aisistent3. Use in chat
In Cursor's AI chat, type:
@aisistent capture the screen and detect buttonsAny MCP Client (generic stdio)
If your MCP client uses stdio transport, the configuration is always the same pattern:
{
"mcpServers": {
"aisistent": {
"command": "aisistent",
"type": "stdio"
}
}
}For HTTP/SSE transport instead of stdio:
# Start AIsistent as an SSE server on port 8100
python -c "from aisistent.server import mcp; mcp.run(transport='sse', port=8100)"Then configure:
{
"mcpServers": {
"aisistent": {
"url": "http://localhost:8100/sse",
"type": "sse"
}
}
}Headless RDP Transport
AIsistent supports two transport modes that can be switched at runtime:
Feature | GUI mode (default) | RDP headless mode |
Local window needed | Yes (Microsoft Remote Desktop) | No |
Capture method |
| Direct RDP framebuffer via |
Click method |
| RDP input channel |
macOS support | Full | Full (no XQuartz needed) |
Linux support | MSS fullscreen | Full |
Windows support | MSS fullscreen | Full |
CLI mode
# GUI mode (default)
aisistent
# RDP headless with inline credentials
aisistent --mode rdp --host 192.168.1.100 --user admin --password secret
# RDP headless with environment variables
export RDP_HOST=192.168.1.100
export RDP_USER=admin
export RDP_PASS=secret
aisistent --mode rdpMCP tools (switch at runtime)
connect_rdp(host="192.168.1.100", username="admin", password="secret")
capture_rdp_screen() # → remote framebuffer, no local window
inject_rdp_click(50, 50) # → click sent via RDP protocol
disconnect_rdp() # → back to GUI modeCredentials precedence
Arguments > Environment variables (RDP_HOST, RDP_USER, RDP_PASS) > Config file
Install
pip install aistent[rdp] # headless RDP only
pip install aistent[all] # everything including RDPwinremote-mcp Integration
AIsistent works alongside winremote-mcp for comprehensive Windows remote management. Run both MCP servers:
aisistent & # AIsistent (stdio)
winremote-mcp --transport sse --port 8100 # winremote-mcp (SSE)AIsistent handles the visual layer (OCR, detection, clicks) while winremote-mcp handles system operations (registry, services, processes, files, etc.).
Docs
Project Structure
AIsistent/
├── aisistent/
│ ├── __init__.py # Version
│ ├── __main__.py # Entry point (argparse: --mode gui|rdp)
│ ├── server.py # MCP server + tools
│ ├── platform.py # OS + device detection
│ ├── config.py # Settings management
│ ├── capture.py # Screen capture (delegates to transport)
│ ├── ocr.py # OCR (Apple Vision / EasyOCR)
│ ├── detection.py # YOLOv8 button detection
│ ├── action.py # Click injection (delegates to transport)
│ ├── benchmark.py # Performance benchmark
│ └── transport/ # Pluggable transport layer
│ ├── __init__.py # get/set transport singleton
│ ├── base.py # Abstract Transport class
│ ├── gui.py # GUI transport (screencapture + pyautogui)
│ └── rdp.py # RDP headless transport (simple-rdp)
├── docs/ # Documentation
├── pyproject.toml
└── README.mdLicense
MIT
🇪🇸 AIsistent
Servidor MCP para automatización RDP no intrusiva. OCR, detección de botones con YOLOv8 e inyección de clics — sin instalar nada en la máquina remota.
Modo GUI (default) | Modo RDP headless | |
Captura | Ventana RDP macOS / MSS pantalla completa | Framebuffer RDP directo (simple-rdp) |
OCR | Apple Vision / EasyOCR | Apple Vision / EasyOCR |
Detección | YOLO (CUDA / MPS / CPU) | YOLO (CUDA / MPS / CPU) |
Click | pyautogui | Canal de input RDP |
Inicio Rápido
# macOS (Apple Silicon)
pip install aistent[apple]
# Windows / Linux (CPU)
pip install aistent[cpu]
# Windows (NVIDIA CUDA)
pip install aistent[cuda]
# RDP headless
pip install aistent[rdp]
aisistent # modo GUI
aisistent --mode rdp --host HOST # modo RDP headlessBenchmark
aisistent-bench # pantallazo real
aisistent-bench --synthetic # imagen sintética
aisistent-bench --image captura.png # imagen propiaResultados reales (MacBook M5 — MPS)
Paso | Tiempo | Elementos |
Captura | ~0.22s | — |
OCR (Apple Vision) | ~0.43s | 102 textos |
YOLO (MPS float16) | ~0.76s | 59 botones |
Total | ~1.4s | — |
Transporte RDP headless
aisistent --mode rdp --host 192.168.1.100 --user admin --password pass
# O vía tool MCP:
# connect_rdp(host="...", username="...", password="...")
# disconnect_rdp()Integración con MCP Clients
Hermes MCP
Añade AIsistent como servidor MCP en ~/.config/hermes/config.json:
{
"mcpServers": {
"aisistent": {
"command": "aisistent",
"type": "stdio"
}
}
}Luego inicia Hermes: hermes
OpenCode
Añade en ~/.config/opencode/opencode.jsonc:
{
"mcpServers": {
"aisistent": {
"command": "aisistent",
"type": "stdio"
}
}
}Claude Desktop
Añade en ~/Library/Application Support/Claude/claude_desktop_config.json:
{
"mcpServers": {
"aisistent": {
"command": "aisistent",
"type": "stdio"
}
}
}Licencia
MIT
Maintenance
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Servers
- Alicense-qualityBmaintenanceMCP server that provides computer control capabilities including mouse movements, keyboard actions, screenshot capture with OCR, and window management through a unified API.Last updated155MIT
- Alicense-qualityDmaintenanceAn MCP server for computer automation that provides tools for screenshots, mouse actions, keyboard input, and drag-and-drop functionality. It supports cross-platform desktop interaction for both Linux (X11) and Windows environments.Last updated5002MIT
- AlicenseAqualityBmaintenanceWindows desktop automation MCP server — screenshot, mouse, keyboard & UI Automation. Lets LLM agents see and control your Windows desktop directly.Last updated301,5918MIT
- Alicense-qualityCmaintenanceGUI automation MCP server that enables AI agents to see and control the Windows desktop using a local Vision LLM (Ollama), supporting screenshot analysis, mouse/keyboard actions, and autonomous task execution.Last updated4MIT
Related MCP Connectors
The MCP server for Azure DevOps, bringing the power of Azure DevOps directly to your agents.
MCP server for Appcircle mobile CI/CD platform.
A paid remote MCP for hosted MCP server, built to return verdicts, receipts, usage logs, and audit-r
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/LaAlquimia/AIsistent'
If you have feedback or need assistance with the MCP directory API, please join our Discord server