LSP and Computer Use

Relevant source files

• agent/file_safety.py
• apps/desktop/src/app/settings/computer-use-panel.tsx
• contributors/emails/dickson.neoh@gmail.com
• tests/agent/lsp/test_shell_linter_lsp_skip.py
• tests/agent/test_file_safety.py
• tests/agent/test_file_safety_credentials.py
• tests/agent/test_file_safety_session_state.py
• tests/computer_use/test_cua_atexit_teardown.py
• tests/computer_use/test_cua_cli_fallback_env.py
• tests/computer_use/test_cua_no_overlay.py
• tests/computer_use/test_cua_spawn_env_sanitization.py
• tests/computer_use/test_doctor.py
• tests/computer_use/test_permissions_resolution.py
• tests/hermes_cli/test_update_zip_symlink_reject.py
• tests/hermes_cli/test_xai_model_flow.py
• tests/tools/test_computer_use.py
• tests/tools/test_computer_use_capture_routing.py
• tests/tools/test_computer_use_cua_backend_linux.py
• tests/tools/test_computer_use_delivery_ladder.py
• tests/tools/test_computer_use_vision_routing.py
• tests/tools/test_container_cwd_sanitize.py
• tests/tools/test_file_operations.py
• tests/tools/test_file_operations_edge_cases.py
• tests/tools/test_file_read_guards.py
• tests/tools/test_file_staleness.py
• tests/tools/test_file_tools.py
• tests/tools/test_file_tools_container_config.py
• tests/tools/test_file_tools_cwd_resolution.py
• tests/tools/test_file_tools_tilde_profile.py
• tests/tools/test_file_write_safety.py
• tests/tools/test_fuzzy_match.py
• tests/tools/test_line_ending_preservation.py
• tests/tools/test_modal_sandbox_fixes.py
• tests/tools/test_patch_failure_tracking.py
• tests/tools/test_patch_parser.py
• tests/tools/test_read_loop_detection.py
• tests/tools/test_resolve_path.py
• tests/tools/test_session_cwd_store.py
• tests/tools/test_terminal_task_cwd.py
• tests/tools/test_write_deny.py
• tools/computer_use/backend.py
• tools/computer_use/cua_backend.py
• tools/computer_use/doctor.py
• tools/computer_use/permissions.py
• tools/computer_use/schema.py
• tools/computer_use/tool.py
• tools/computer_use/vision_routing.py
• tools/file_operations.py
• tools/file_tools.py
• tools/fuzzy_match.py
• tools/patch_parser.py

This page documents two advanced capabilities within the Hermes Agent framework that bridge the gap between text-based reasoning and environment-aware execution:

1. Language Server Protocol (LSP) Integration: The system that wires semantic diagnostics (errors, warnings, hints) from language servers into file modification tools.
2. Computer Use: Universal desktop control (macOS, Windows, Linux) via cua-driver, allowing agents to interact with GUI applications in the background.

Language Server Protocol (LSP) Integration

The Hermes Agent integrates with Language Servers (e.g., pyright, gopls, rust-analyzer) to provide semantic feedback directly to the agent after it modifies code. This feedback loop allows the agent to detect and fix syntax errors or type mismatches immediately after applying a change.

Integration with File Tools

LSP diagnostics are integrated into the core file manipulation tools: write_file and patch. When these tools execute, they return a result object that includes a dedicated lsp_diagnostics field tools/file_operations.py182-188

• WriteResult: Captures diagnostics after a full file write. It surfaces semantic errors as a separate signal from traditional linting tools/file_operations.py177-189
• PatchResult: Captures diagnostics after a partial update (hunk-based). It allows the model to see syntax and semantic errors introduced by the specific edit tools/file_operations.py195-207

Data Flow and Semantic Feedback

The integration ensures that the agent doesn't just "write and pray." Instead, it receives structured signals from the compiler/linter. The ShellFileOperations class in tools/file_operations.py wraps the terminal backend's execution interface to provide this unified API tools/file_operations.py8-16

LSP Semantic Diagnostic Flow

Sources: <FileRef file-url="https://github.com/NousResearch/hermes-agent/blob/af8d698b/tools/file_operations.py#L8-L16" min=8 max=16 file-path="tools/file_operations.py">Hii</FileRef>, <FileRef file-url="https://github.com/NousResearch/hermes-agent/blob/af8d698b/tools/file_operations.py#L182-L188" min=182 max=188 file-path="tools/file_operations.py">Hii</FileRef>, <FileRef file-url="https://github.com/NousResearch/hermes-agent/blob/af8d698b/tools/file_operations.py#L195-L207" min=195 max=207 file-path="tools/file_operations.py">Hii</FileRef>

Robust Matching and Fuzzy Logic

To ensure patches apply correctly even when LLMs generate slightly incorrect whitespace or indentation, Hermes uses a multi-strategy fuzzy matching chain in tools/fuzzy_match.py tools/fuzzy_match.py3-9

• Strategies: The 9-strategy chain includes exact, line_trimmed, whitespace_normalized, indentation_flexible, escape_normalized, trimmed_boundary, unicode_normalized, block_anchor, and context_aware tools/fuzzy_match.py73-83
• Escape Drift Guard: Detects if the transport layer added spurious backslashes (e.g., \' instead of ') during tool-call serialization and blocks the write with a helpful error tools/fuzzy_match.py96-109
• Indentation Preservation: When a non-exact strategy matches, the replacement text is automatically re-indented to match the file's actual indentation depth tools/fuzzy_match.py111-115

Sources: <FileRef file-url="https://github.com/NousResearch/hermes-agent/blob/af8d698b/tools/fuzzy_match.py#L3-L15" min=3 max=15 file-path="tools/fuzzy_match.py">Hii</FileRef>, <FileRef file-url="https://github.com/NousResearch/hermes-agent/blob/af8d698b/tools/fuzzy_match.py#L73-L83" min=73 max=83 file-path="tools/fuzzy_match.py">Hii</FileRef>, <FileRef file-url="https://github.com/NousResearch/hermes-agent/blob/af8d698b/tools/fuzzy_match.py#L96-L115" min=96 max=115 file-path="tools/fuzzy_match.py">Hii</FileRef>

Safety and Deny Lists

Even when using LSP-enabled tools, the agent is subject to a strict write-deny list and path guards.

• Denied Paths: Includes sensitive files like .ssh/authorized_keys, .ssh/id_rsa, .netrc, and various credential files tools/file_operations.py45-53 tests/tools/test_file_operations.py29-41
• Credential Protection: Access to .anthropic_oauth.json, mcp-tokens/, and pairing/ directories within HERMES_HOME is explicitly blocked to prevent exfiltration or unauthorized access escalation tests/tools/test_file_operations.py65-82
• Safe Root Sandboxing: The HERMES_WRITE_SAFE_ROOT environment variable can be used to sandbox writes to specific subtrees agent/file_safety.py83-98
• Read-loop Detection: The _read_tracker in tools/file_tools.py detects redundant reads, warning the agent on the 3rd consecutive identical read and blocking the 4th to prevent infinite loops tests/tools/test_read_loop_detection.py5-13 tests/tools/test_read_loop_detection.py85-105

Sources: <FileRef file-url="https://github.com/NousResearch/hermes-agent/blob/af8d698b/tools/file_operations.py#L45-L53" min=45 max=53 file-path="tools/file_operations.py">Hii</FileRef>, <FileRef file-url="https://github.com/NousResearch/hermes-agent/blob/af8d698b/tests/tools/test_file_operations.py#L29-L82" min=29 max=82 file-path="tests/tools/test_file_operations.py">Hii</FileRef>, <FileRef file-url="https://github.com/NousResearch/hermes-agent/blob/af8d698b/agent/file_safety.py#L83-L98" min=83 max=98 file-path="agent/file_safety.py">Hii</FileRef>, <FileRef file-url="https://github.com/NousResearch/hermes-agent/blob/af8d698b/tests/tools/test_read_loop_detection.py#L5-L115" min=5 max=115 file-path="tests/tools/test_read_loop_detection.py">Hii</FileRef>

---

Computer Use (Universal Desktop Control)

The computer_use tool enables universal desktop control across macOS, Windows, and Linux. It uses cua-driver to interact with the OS in the background via accessibility APIs tools/computer_use/cua_backend.py1-11

Backend Architecture

The tool is model-agnostic and provides a standardized interface for interacting with the desktop environment.

• cua-driver: The primary backend. It facilitates communication between the agent and the operating system via MCP (Model Context Protocol) over stdio tools/computer_use/cua_backend.py26-34
• Universal Schema: Hermes uses a standard OpenAI function-calling schema (COMPUTER_USE_SCHEMA) so every tool-capable model can drive it, avoiding vendor-specific native types tools/computer_use/schema.py1-7 tests/tools/test_computer_use.py41-58
• Multimodal Output: Actions that capture the screen return multi-part tool messages containing text summaries and base64-encoded PNG images tools/computer_use/tool.py19-30

Interaction Model

The tool supports various modes of interaction:

1. Coordinate-based: Direct interaction with specific screen coordinates [x, y] tools/computer_use/schema.py131-141
2. Vision-assisted (SOM): Using "Set-of-Mark" (SOM) numbered overlays on interactable elements, allowing the model to click by element index tools/computer_use/schema.py62-69 tools/computer_use/schema.py123-130
3. AX (Accessibility): Utilizing the text-based representation of the accessibility tree (no image) tools/computer_use/schema.py67-69

Computer Use Tool Execution Flow

Sources: <FileRef file-url="https://github.com/NousResearch/hermes-agent/blob/af8d698b/tools/computer_use/tool.py#L1-L12" min=1 max=12 file-path="tools/computer_use/tool.py">Hii</FileRef>, <FileRef file-url="https://github.com/NousResearch/hermes-agent/blob/af8d698b/tools/computer_use/cua_backend.py#L1-L18" min=1 max=18 file-path="tools/computer_use/cua_backend.py">Hii</FileRef>, <FileRef file-url="https://github.com/NousResearch/hermes-agent/blob/af8d698b/tools/computer_use/cua_backend.py#L38-L50" min=38 max=50 file-path="tools/computer_use/cua_backend.py">Hii</FileRef>, <FileRef file-url="https://github.com/NousResearch/hermes-agent/blob/af8d698b/tests/tools/test_computer_use.py#L41-L50" min=41 max=50 file-path="tests/tools/test_computer_use.py">Hii</FileRef>

Safety and Approval

Computer use is subject to safety checks and human-in-the-loop approval for destructive actions.

• Action Classification: Actions like capture, wait, and list_apps are classified as _SAFE_ACTIONS tools/computer_use/tool.py80 while click, type, and key are _DESTRUCTIVE_ACTIONS tools/computer_use/tool.py83-86 and require approval via set_approval_callback tools/computer_use/tool.py68-76
• Blocked Key Combos: Critical system shortcuts like "Log Out" (cmd+shift+q), "Lock Screen" (cmd+ctrl+q), or "Empty Trash" are hard-blocked in _BLOCKED_KEY_COMBOS tools/computer_use/tool.py90-103
• Text Injection Guard: The type action is scanned via _is_blocked_type for dangerous shell patterns like curl | bash or sudo rm -rf tools/computer_use/tool.py118-133
• No-Overlay Policy: To save CPU, Hermes can suppress the cua-driver cursor overlay using the --no-overlay flag, which is auto-detected for macOS and headless Linux/WSL2 tests/computer_use/test_cua_no_overlay.py1-11 tests/computer_use/test_cua_no_overlay.py19-49

Sources: <FileRef file-url="https://github.com/NousResearch/hermes-agent/blob/af8d698b/tools/computer_use/tool.py#L68-L103" min=68 max=103 file-path="tools/computer_use/tool.py">Hii</FileRef>, <FileRef file-url="https://github.com/NousResearch/hermes-agent/blob/af8d698b/tools/computer_use/tool.py#L118-L133" min=118 max=133 file-path="tools/computer_use/tool.py">Hii</FileRef>, <FileRef file-url="https://github.com/NousResearch/hermes-agent/blob/af8d698b/tests/computer_use/test_cua_no_overlay.py#L1-L49" min=1 max=49 file-path="tests/computer_use/test_cua_no_overlay.py">Hii</FileRef>

Summary of Actions

Action	Description	Targeting
capture	Takes a screenshot and/or retrieves UI state.	som, vision, ax modes
click	Performs a mouse click (left, right, double, middle).	Coordinates/Elements
type	Types a string of text into the active field.	String
key	Presses specific key combinations (e.g., "cmd+space").	Key chords
list_apps	Lists currently running applications.	N/A
focus_app	Brings a specific application to the foreground.	App Name
drag	Performs a drag-and-drop operation.	Coordinates/Elements
scroll	Scrolls the view.	Direction/Amount
set_value	Directly sets value for sliders or popups.	Element Index

Sources: <FileRef file-url="https://github.com/NousResearch/hermes-agent/blob/af8d698b/tools/computer_use/tool.py#L80-L86" min=80 max=86 file-path="tools/computer_use/tool.py">Hii</FileRef>, <FileRef file-url="https://github.com/NousResearch/hermes-agent/blob/af8d698b/tools/computer_use/schema.py#L32-L56" min=32 max=56 file-path="tools/computer_use/schema.py">Hii</FileRef>, <FileRef file-url="https://github.com/NousResearch/hermes-agent/blob/af8d698b/tests/tools/test_computer_use.py#L67-L73" min=67 max=73 file-path="tests/tools/test_computer_use.py">Hii</FileRef>