Voice and TTS

Relevant source files

• tests/gateway/test_internal_event_bypass_pairing.py
• tests/run_agent/test_24996_fallback_exhaustion_cooldown.py
• tests/tools/conftest.py
• tests/tools/test_approved_command_clean_slate.py
• tests/tools/test_code_execution.py
• tests/tools/test_code_execution_windows_env.py
• tests/tools/test_file_tools_live.py
• tests/tools/test_interrupt.py
• tests/tools/test_local_env_blocklist.py
• tests/tools/test_local_env_cwd_recovery.py
• tests/tools/test_local_env_windows_msys.py
• tests/tools/test_notify_on_complete.py
• tests/tools/test_process_registry.py
• tests/tools/test_tirith_security.py
• tests/tools/test_transcription.py
• tests/tools/test_transcription_dotenv_fallback.py
• tests/tools/test_transcription_tools.py
• tests/tools/test_tts_command_providers.py
• tests/tools/test_tts_deepinfra.py
• tests/tools/test_tts_dotenv_fallback.py
• tests/tools/test_tts_mistral.py
• tests/tools/test_tts_piper.py
• tests/tools/test_voice_cli_integration.py
• tests/tools/test_voice_mode.py
• tests/tools/test_watch_patterns.py
• tools/code_execution_tool.py
• tools/environments/local.py
• tools/interrupt.py
• tools/process_registry.py
• tools/tirith_security.py
• tools/transcription_tools.py
• tools/tts_tool.py
• tools/voice_mode.py
• tools/xai_http.py
• website/docs/user-guide/features/tts.md

This page provides a high-level overview of the voice interaction and speech synthesis capabilities of Hermes Agent. The system supports full-duplex voice conversations in the CLI, automated transcription of voice messages on messaging platforms (Telegram, Discord, WhatsApp, Slack, Signal, etc.), and high-quality text-to-speech (TTS) responses across multiple providers.

Overview of Voice Capabilities

Hermes Agent bridges the gap between natural language speech and the agentic core through a modular pipeline of Speech-to-Text (STT) and Text-to-Speech (TTS) components.

Feature	Description	Implementation
Voice Mode	Real-time, push-to-talk or continuous voice interaction in the CLI and Discord.	tools/voice_mode.py tools/voice_mode.py1-10
STT Pipeline	Multi-provider transcription (Local, Groq, OpenAI, Mistral, xAI, ElevenLabs).	tools/transcription_tools.py tools/transcription_tools.py5-15
TTS Synthesis	Multi-provider synthesis (Edge, ElevenLabs, OpenAI, MiniMax, Mistral, Gemini, xAI, NeuTTS, KittenTTS, Piper).	tools/tts_tool.py tools/tts_tool.py5-15
Platform Delivery	Integration with Discord Voice Channels and messaging voice bubbles.	`website/docs/user-guide/features/tts.md:32-40]()

System Architecture Flow

The following diagram illustrates how audio data flows from a user into the AIAgent and back out as synthesized speech, bridging the "Natural Language Space" to the "Code Entity Space".

Audio Processing Pipeline

Sources: tools/voice_mode.py1-10 tools/transcription_tools.py23-28 tools/tts_tool.py32-35

---

Voice Mode

Voice Mode allows for a hands-free interaction experience within the terminal and integrated messaging platforms. It utilizes sounddevice and numpy for low-latency audio I/O and implements environment detection to ensure compatibility across SSH, Docker, and WSL tools/voice_mode.py141-190

• Activation: Controlled via /voice on in the CLI. The system uses a _voice_lock to protect state changes during concurrent recording and processing.
• Silence Detection: A two-stage algorithm confirms speech (RMS threshold > 200) and detects the end of an utterance after 3.0 seconds of silence.
• Environment Support: Includes specialized support for Termux via termux-microphone-record and termux-api when PortAudio is unavailable tools/voice_mode.py61-85
• SSH/Docker Connectivity: Detects reachable PulseAudio or PipeWire sockets (e.g., pulse/native or pipewire-0) to allow audio forwarding over remote connections tools/voice_mode.py89-139
• Discord Integration: The Discord adapter supports voice channels, requiring discord.py[voice] and PyNaCl for encryption and Opus bindings.

For details, see Voice Mode.

Sources: tools/voice_mode.py1-10 tools/voice_mode.py141-190 tools/voice_mode.py89-139

---

TTS and Transcription

The TTS and STT systems are designed for "Zero Config" operation while supporting premium cloud providers for enhanced quality.

Speech-to-Text (STT)

The transcription system resolves providers in a priority chain: Local (faster-whisper) > Groq > OpenAI > Mistral > xAI > ElevenLabs tools/transcription_tools.py86-93

• Local: Uses faster-whisper to run models locally or executes a shell command defined in HERMES_LOCAL_STT_COMMAND tools/transcription_tools.py94-95
• Validation: All audio files are validated for format (mp3, ogg, wav, etc.) and size (max 25MB) before being sent to providers tools/transcription_tools.py104-106
• Dotenv Integration: STT providers consult get_env_value() to ensure keys in ~/.hermes/.env are honored tools/transcription_tools.py51-63

Text-to-Speech (TTS)

The text_to_speech_tool handles synthesis and automatic conversion. It utilizes ffmpeg to convert various outputs to OGG Opus format specifically for Telegram voice bubbles tools/tts_tool.py24-26

TTS Provider Mapping

Provider	Code Entity	Key Features
Edge TTS	_import_edge_tts	Free, Microsoft Edge neural voices tools/tts_tool.py86-96
ElevenLabs	_import_elevenlabs	Premium quality, model-aware character caps tools/tts_tool.py98-118
OpenAI	_import_openai_client	High quality, gpt-4o-mini-tts support tools/tts_tool.py119-122
MiniMax TTS	DEFAULT_MINIMAX_MODEL	High-quality with voice cloning tools/tts_tool.py179-181
Mistral	_import_mistral_client	Multilingual, native Opus support tools/tts_tool.py124-141
Google Gemini	DEFAULT_GEMINI_TTS_MODEL	30 prebuilt voices, persona prompt support website/docs/user-guide/features/tts.md68-72
xAI TTS	DEFAULT_XAI_VOICE_ID	Grok voices, uses xAI Grok OAuth tools/tts_tool.py184-186
NeuTTS	DEFAULT_PROVIDER	Local, free, on-device TTS tools/tts_tool.py13
KittenTTS	_import_kittentts	On-device 25MB nano model tools/tts_tool.py148-151
Piper	_import_piper	Local neural VITS, 44 languages tools/tts_tool.py154-163

Voice/TTS Class Association

Sources: tools/transcription_tools.py23-28 tools/tts_tool.py5-15 tools/voice_mode.py32-50

xAI Integration and Multimodal Storage

The voice and xAI systems overlap in multimodal capabilities. The resolve_xai_http_credentials helper ensures unified authentication for both TTS and multimodal generation. Additionally, the system manages the persistence of generated media (such as audio or video files) on xAI's infrastructure with configurable expiry.

For details, see TTS and Transcription.

Sources: tools/tts_tool.py5-16 tools/transcription_tools.py5-16 tools/voice_mode.py1-10 tools/transcription_tools.py51-63