This document covers the tools that enable web interaction, browser automation, and visual analysis: web search and extraction tools/web_tools.py10-14 automated browser control tools/browser_tool.py5-9 image understanding tools/vision_tools.py9-11 and multimodal video analysis. These tools leverage the auxiliary client pattern agent/auxiliary_client.py51-55 to perform side tasks like summarization and vision analysis without bloating the main conversation context.
The following diagram bridges the natural language tool definitions used by the agent to the underlying code entities and external service backends.
Natural Language to Code Entity Mapping
Sources: tools/web_tools.py11-13 tools/browser_tool.py40-50 tools/vision_tools.py10-11 plugins/web/firecrawl/provider.py53-60 tools/browser_tool.py130-132 tools/web_tools.py168-170
Web tools provide search and extraction capabilities through a pluggable provider system. Backend selection is determined by web.backend in config.yaml or auto-detected from environment variables tools/web_tools.py123-140
The system supports multiple backends, many of which have been migrated to the plugins/web/ directory for better modularity tools/web_tools.py46-50
| Backend | Capabilities | Implementation |
|---|---|---|
| Firecrawl | Search, Extract | plugins/web/firecrawl/provider.py53-60 |
| Parallel | Search, Extract | plugins/web/parallel/provider.py71-74 |
| Tavily | Search, Extract | plugins/web/tavily/provider.py63-67 |
| Exa | Search, Extract | plugins/web/exa/provider.py75 |
| SearXNG | Search | plugins/web/searxng/provider.py27 |
| DuckDuckGo | Search | plugins/web/ddgs/provider.py25 |
Sources: tools/web_tools.py14-19 tools/web_tools.py171-172 tools/web_tools.py123-140
web_search_tool: Dispatches to the resolved provider's search method tools/web_tools.py11web_extract_tool: Extracts content from specific web pages tools/web_tools.py12 It uses intelligent content extraction to create markdown summaries and reduce token usage tools/web_tools.py21-22async_is_safe_url to prevent SSRF by blocking requests to private or internal network addresses tools/web_tools.py100 It also detects sensitive query parameters (like api_key or token) to avoid accidental credential leakage tools/url_safety.py136-155The browser tool provides automated browser control using multiple backends, including cloud services and local Chromium instances tools/browser_tool.py5-9
Sources: tools/browser_tool.py42-50 tools/browser_tool.py130-132 tools/browser_camofox.py114-130
ariaSnapshot for text-based page representation, ideal for LLMs without vision capabilities tools/browser_tool.py11-12node, npx, and agent-browser. Fallback directories are defined in _SANE_PATH_DIRS tools/browser_tool.py156-168_build_browser_env tools/browser_tool.py90-105CAMOFOX_URL is set tools/browser_camofox.py8-9 It includes a "managed persistence" mode to map stable user IDs to persistent browser profile directories tools/browser_camofox.py175-184The vision_analyze_tool handles multimodal image understanding tools/vision_tools.py10
ThreadPoolExecutor sized to the host's core count tools/vision_tools.py98-108 tools/vision_tools.py170-174async_call_llm to route the base64-encoded image to vision-capable models tools/vision_tools.py43The video_analyze_tool provides multimodal video understanding for files and URLs.
vision_analyze calls across extracted frames. The tool is governed by auxiliary vision configuration for timeouts and model selection tools/vision_tools.py53-71Sources: tools/vision_tools.py10-18 tools/vision_tools.py77 tools/vision_tools.py126-144 tools/vision_tools.py170-174
The system provides generation capabilities via provider plugins, primarily focusing on high-performance backends.
| Feature | Description |
|---|---|
| Image Generation | Supports multiple providers including FAL.ai and local backends. |
| Video Generation | Integrated via plugins such as xAI. |
| Managed Gateway | Supports routing through a managed tool gateway for subscribers tools/web_tools.py7-9 |
Sources: tools/web_tools.py7-9
The system includes robust audio processing tools for speech-to-text (STT) and text-to-speech (TTS).
Provides speech-to-text with multiple providers including local (faster-whisper), Groq, OpenAI, Mistral, xAI, and ElevenLabs.
Supports multiple providers including Edge TTS (default), ElevenLabs, OpenAI, and local options like NeuTTS, KittenTTS, and Piper.
config.yaml.Refresh this wiki
This wiki was recently refreshed. Please wait 4 days to refresh again.