The Buzz of S2S — Part 2: Architecture of Understanding — How Speech-to-Speech Models Actually Work

If Part 1 was about why Speech-to-Speech (S2S) models matter, this part is about how they work.

It’s easy to think S2S models are just smarter text-to-speech engines. They aren’t. They are multimodal cognitive systems that compress, translate, and regenerate human voice in real time — while retaining semantics, emotion, and identity.

Let’s open the hood.

🧩 The Evolution from Pipeline to Fusion

For decades, speech systems worked in three disconnected stages:

1. ASR (Automatic Speech Recognition) — convert speech → text
2. NLP/LLM reasoning — interpret text → meaning
3. TTS (Text-to-Speech) — generate speech → audio

It worked, but like a chain of translators passing notes down the line, something was always lost — tone, rhythm, latency, empathy.

End-to-end S2S models fuse those three into one neural system. Instead of converting everything to words, they operate in a latent acoustic-semantic space — a sort of “thought-language” between speech and meaning.

This fusion does two things:

• Removes bottlenecks like text tokenization, which loses emotion and timing.
• Adds continuity — emotion and pacing flow naturally through the model.

⚙️ The Three Pillars of S2S Architecture

Modern S2S systems typically consist of three core modules — though in the best models, the boundaries blur:

1. Speech Encoder — Hearing the World

The encoder transforms a waveform into a compressed vector space — an embedding that captures what is being said and how it’s being said.

Press enter or click to view image in full size

Techniques:

• Spectrogram conversion: Transforming raw waveforms into mel-spectrograms.
• Self-supervised feature extraction: Using models like wav2vec 2.0, HuBERT, or Whisper’s encoder backbone to learn phonetic, tonal, and emotional cues.
• Temporal masking: To make encoders robust to noise and missing segments.

Good encoders create embeddings dense enough to represent both content and style — the same sentence whispered or shouted must map differently.

Think of it as “compressing your voice into thought vectors.”

2. Latent Representation — Thinking in Voice

Between encoder and decoder lies the semantic bottleneck, often a high-dimensional latent space trained with contrastive learning or transformer attention.

This is where the magic happens:

• The model aligns acoustic similarity with semantic meaning.
• Multilingual embeddings allow cross-language transfer — “bonjour” and “hello” sit close in this space.
• Some systems (like Meta’s SeamlessM4T) build shared embeddings for text and speech, enabling direct speech-to-speech translation.

This layer behaves like a brain’s “inner voice” — not yet sound, not yet words, but something in between.

3. Speech Decoder — Speaking It Back

Press enter or click to view image in full size

The decoder (or vocoder) reconstructs high-fidelity speech from latent vectors.

Traditional vocoders like WaveNet or WaveRNN used autoregressive sampling — beautiful but slow.
Modern ones like HiFi-GAN, BigVGAN, and StyleTTS2 can generate speech in real time, preserving timbre and expressive inflection.

The decoder may also condition on:

• Speaker embedding: who is speaking
• Emotion or prosody vector: how it’s spoken
• Language tag: which language to produce

This is how Google Translatotron 2 can keep your voice identity consistent even across languages.

🧠 Transformers and Diffusion: The Core Engines

Press enter or click to view image in full size

The transformer architecture — first used in text with Attention Is All You Need — underpins most S2S models.

• Encoder-decoder transformers handle temporal dependencies in speech better than RNNs ever could.
• Self-attention learns which parts of an utterance matter most when predicting the next acoustic frame.
• Diffusion-based decoders (used in newer research from Meta and Cartesia) model the stochastic nature of sound, producing speech that feels organic and lifelike.

Some hybrid models use Neural Audio Codecs like SoundStream or EnCodec, compressing raw audio into discrete tokens — think of them as the “subwords of sound.”
These tokens can be directly processed by language models like GPT-style transformers, turning voice into a symbolic space for reasoning.

⚡ The Shift to Real-Time S2S

Real-time S2S is a race between compute efficiency and fidelity.

• OpenAI’s GPT-4o Voice achieves ~200–300 ms response latency using highly optimized streaming transformers.
• Cartesia’s Sonic pipeline achieves similar speed with smaller, quantized models and a proprietary neural codec that compresses speech 30× faster than Whisper-scale encoders.
• Google’s Translatotron 2 runs in near-real-time translation while preserving speaker characteristics.

These systems use chunked inference, processing partial segments of speech before the full sentence finishes — like humans anticipating meaning mid-conversation.

🧮 Cross-Lingual and Emotional Control

The holy grail of S2S is not just translation — it’s emotionally and linguistically faithful voice transfer.

• Meta’s SeamlessM4T integrates code-switching (mixing languages within the same sentence).
• Microsoft’s VALL-E achieves zero-shot emotion transfer — replicate the style, pitch, and energy of a voice sample without retraining.
• Cartesia adds dynamic prosody conditioning, so the same output can sound calm, excited, or empathetic depending on the conversation.

Emotion vectors, often extracted via prosody encoders, represent features like intonation, duration, and loudness — a growing field of affective AI.

🧱 Startups Redefining Architecture

Cartesia

Sonic uses diffusion-based neural codecs for sub-300 ms latency and expressive realism. Its encoder-decoder pair is trained to maintain paralinguistic features — laughter, breath, tone — often lost in traditional vocoders.

Decagon

Focuses on secure deployment of S2S within enterprise stacks. Its architecture integrates speech anonymization layers and on-prem inference modules, allowing organizations to use S2S without leaking identity data.

Cresta

Combines LLM reasoning with voice-stream models for real-time coaching in contact centers. Their hybrid architecture fuses speech recognition streams with LLM embeddings, enabling actionable AI feedback while the conversation is still happening.

ServiceNow

Is embedding S2S into its Now Assist agent ecosystem — effectively creating voice-first digital employees. Their stack leverages external speech models (like Whisper derivatives) but wraps them in enterprise governance, context-awareness, and workflow integration.

🧬 Putting It Together

An end-to-end Speech-to-Speech model today looks something like this:

Incoming Speech
     ↓
[Speech Encoder]  → Latent Acoustic Representation →  [Speech Decoder]
     ↓                                         ↓
 Self-Supervised Features              Conditional Voice Reconstruction
     ↓                                         ↓
 Emotion, Speaker, Language           Real-time, Natural Output Speech

Between these stages, transformer attention layers and diffusion decoders create the sense of continuity, personality, and empathy that defines a human-like conversation.

🚀 Why It Matters

The evolution from discrete pipelines to end-to-end fusion is not just technical — it’s philosophical.

It signals the birth of speech-native intelligence: systems that perceive and communicate the way humans do.
Not through strings of text, but through shared sound, rhythm, and expression.

🧭 Coming Up Next

In Part 3, we’ll compare the Titans and Innovators head-to-head:

• OpenAI’s GPT-4o Voice
• Google’s Translatotron 2 and AudioPaLM
• Meta’s SeamlessM4T & Voicebox
• Microsoft’s VALL-E
• Cartesia, Decagon, Cresta, and ServiceNow

We’ll analyze architecture, latency, emotional fidelity, and strategy — who’s building for research, who’s building for the enterprise, and who’s building for the next generation of human-AI dialogue.

When machines begin to understand us not as text, but as tone — that’s not just communication; that’s connection.