Voice Mode provides real-time audio interaction capabilities across the CLI and messaging gateways. It encompasses local microphone capture, speech-to-text (STT) transcription, text-to-speech (TTS) synthesis, and specialized integration with Discord voice channels.
The voice system is divided into two primary execution contexts: the CLI Voice Mode (push-to-talk/continuous) and the Gateway Voice Mode (asynchronous voice messages and live Discord VC).
The following diagram illustrates how raw audio is captured, processed, and delivered to the AIAgent.
Voice Input Pipeline
Sources: tools/voice_mode.py1-10 tools/voice_mode.py61-64 tools/voice_mode.py22
Voice mode in the messaging gateway allows the agent to process voice notes on platforms like Telegram, Slack, and Discord.
Users can control voice behavior using the /voice command within messaging platforms. These settings allow toggling between text and audio responses. The GatewayRunner manages these settings, persisting them to a JSON file (gateway_voice_mode.json) and synchronizing them with platform adapters tests/gateway/test_voice_command.py77-83 tests/gateway/test_voice_command.py149-160
| Command | Mode | Effect |
|---|---|---|
/voice on | voice_only | Agent responds with voice only |
/voice tts | all | Agent responds with both text and voice |
/voice off | off | Voice replies disabled |
/voice status | N/A | Returns the current voice configuration for the chat |
The _handle_voice_command method in GatewayRunner processes these commands, updates the _voice_mode dictionary, and saves the state tests/gateway/test_voice_command.py98-146 The _sync_voice_mode_state_to_adapter method ensures that platform adapters (e.g., TelegramAdapter) have their _auto_tts_disabled_chats and _auto_tts_enabled_chats sets correctly populated based on the persisted voice mode settings tests/gateway/test_voice_command.py162-189
The SlackAdapter and other adapters manage incoming media including audio files.
cache_document_from_bytes and cache_video_from_bytes to store incoming voice notes before processing plugins/platforms/slack/adapter.py58-59files:read scope is required to download voice notes and audio website/docs/user-guide/messaging/slack.md96Messaging Voice Integration Class Space
Sources: plugins/platforms/slack/adapter.py42-60 website/docs/user-guide/messaging/slack.md81-98 website/docs/user-guide/messaging/discord.md128-140 tests/gateway/test_voice_command.py77-83 tests/gateway/test_voice_command.py149-189
Transcription is handled by the transcribe_audio utility. The system supports a provider resolution chain including Local (faster-whisper), Groq, and OpenAI.
The system filters Whisper's phantom text (e.g., "Thank you for watching") using a set of known hallucination phrases and regex patterns to catch repetitive variations.
TTS synthesis supports multiple backends including ElevenLabs, OpenAI, and NeuTTS.
**), italics (*), and headers (#) to ensure the synthesizer only receives speakable text. This is handled by functions like _strip_mdv2 in Telegram adapter tests/gateway/test_telegram_format.py38-43format_message function in platform adapters handles escaping and preserving code blocks for display, but for TTS, these are typically stripped tests/gateway/test_telegram_format.py119-180NeuTTS provides a free, local synthesis option using espeak-ng as a phonemizer backend.The AudioRecorder logic in tools/voice_mode.py manages local hardware via sounddevice and numpy.
detect_audio_environment checks for SSH, Docker, and WSL (PulseAudio) sockets to ensure hardware access tools/voice_mode.py142-192_pulse_socket_reachable validates PULSE_SERVER and XDG_RUNTIME_DIR sockets (e.g., checking pulse/native or pipewire-0) to support audio forwarding over SSH or into containers tools/voice_mode.py89-139 tests/tools/test_voice_mode.py98-114termux-microphone-record via _termux_microphone_command and checks for the com.termux.api package tools/voice_mode.py61-86_import_audio() to prevent crashes in headless environments where PortAudio might be missing tools/voice_mode.py32-41CLI Voice Implementation Entity Space
Sources: tools/voice_mode.py32-48 tools/voice_mode.py61-86 tools/voice_mode.py89-139 tools/voice_mode.py142-192 tests/tools/test_voice_mode.py75-130
Refresh this wiki
This wiki was recently refreshed. Please wait 3 days to refresh again.