How AI Converts Speech into Text: MFCC, Mel Spectrogram, Acoustic Models & Language Models
A Simple Guide to Audio Features & ASR
Audio Feature Extraction
What Is Feature Extraction?
Before a computer can understand speech, it needs to convert your voice into a format it can process. Raw audio is just thousands of numbers per second — too much, too noisy, and full of irrelevant data. Feature extraction squeezes this down to only the key patterns that the ASR engine needs.
Key Idea
A computer cannot “hear” audio the way humans do. Feature extraction summarises the sound into meaningful patterns — like describing a painting in a few words instead of listing every pixel.
The Feature Extraction Pipeline
Here is the step-by-step process audio goes through:
STEP 1 : Break Audio into Small Chunks (Framing)
The audio stream is sliced into tiny overlapping windows — each about 25 milliseconds long. Each chunk is short enough that the sound stays stable within it. Chunks overlap by 15 ms so no sound is lost at the edges.
STEP 2 : Smooth the Edges (Windowing)
The edges of each chunk are tapered gradually to zero. This prevents artificial frequencies from appearing when the chunk is analysed — like fading a photo at the edges instead of cutting it sharply.
STEP 3 : Find the Frequencies (FFT)
A mathematical operation (Fast Fourier Transform) analyses each chunk and tells us: how much of each frequency is present? Think of it like a music equaliser — showing how much bass, mid, and treble exist at that moment.
STEP 4 : Match Human Hearing (Mel Filterbank)
Humans hear low frequencies more clearly than high ones. The Mel scale adjusts the output to match how our ears actually work — stretching low frequencies and compressing high ones.
STEP 5A : Mel Spectrogram — Stop Here for Modern AI
The result is a 2D picture of sound: time on one axis, frequency on the other. Neural networks (like Whisper) use this directly. It keeps 80–128 values per frame.
STEP 5B : MFCC — One More Compression Step
Apply DCT (Discrete Cosine Transform) to compress the 80+ values down to just 13–40 numbers. This is MFCC — very compact, used by traditional systems and edge devices.
MFCC vs Mel Spectrogram
Both formats represent the same audio, but in different levels of detail. The right choice depends on your system:
MFCC = describing a painting in 3 words (‘blue, bright, calm’). Mel Spectrogram = a thumbnail of the same painting. The thumbnail keeps more detail; the 3 words are smaller and faster to process.
Visual Comparison: Mel Spectrogram vs MFCC
The ASR Engine (ACMOD + LM)
Once audio features are ready, the ASR engine takes over. It has two components working together to turn those numbers into words:
Acoustic Model (ACMOD)
Listens to the audio features and says: “I heard these sounds.” Maps features to phonemes — the basic units of sound.
Language Model (LM)
Get VoiceAIEngineer’s stories in your inbox
Join Medium for free to get updates from this writer.
Reads those sounds and says: “Given the context, these sounds most likely form this word.” Resolves ambiguity.
Step 1 — Acoustic Model: Hearing the Sounds
What Are Phonemes?
Phonemes are the smallest units of sound in a language. English has about 44. Every word is made from a sequence of phonemes. For example:
Think of the Acoustic Model as someone who is hard of hearing — they catch the sounds but sometimes mishear similar ones (‘set’ vs ‘sat’). The Language Model will sort it out next.
Step 2 — Language Model: Picking the Right Words
Phonemes alone are ambiguous. The sounds /t u/ could mean ‘to’, ‘two’, or ‘too’. The Language Model looks at all possible words and picks the sequence that makes the most sense in context.
Imagine your friend texts you: ‘I need too apples.’ You instantly know they meant ‘two’ because ‘too apples’ makes no grammatical sense. The Language Model does exactly this — it knows which word fits the context.
For Smart Home Commands
A Language Model trained on smart home commands knows these patterns:
• ‘set [device] to [number] [unit]’ is a common command structure
• After ‘AC’, words like ‘degrees’ and ‘temperature’ are highly likely
• ‘acne’ or ‘acknowledge’ would score very low in this context
Step 3 — The Decoder: Combining ACMOD + LM
The decoder runs both models together using Beam Search. Instead of guessing one word at a time, it keeps several possible sentences alive and scores each one, then picks the winner.
Full Pipeline — End to End
Step-by-Step Walkthrough
Here is the complete journey from your voice to the executed command “set AC to 22 degrees”:
1. You say: “set AC to 22 degrees”
2. Microphone records your voice as 16,000 numbers per second
3. Audio is sliced into 25ms overlapping frames
4. FFT finds the frequencies present in each frame
5. Mel filterbank converts frequencies to the human hearing scale
6. Output: Mel Spectrogram (80 values/frame) or MFCC (13–40 values/frame)
7. Acoustic Model analyses each frame and finds the most likely phonemes
8. Decoder runs Beam Search — keeps the top candidate sentences alive simultaneously
9. Language Model scores each candidate — ‘set AC to 22 degrees’ wins
10. Final transcript produced: “set AC to 22 degrees”
11. Intent Parser extracts: action=SET | device=AC | value=22 | unit=DEGREES
12. Command executed — AC temperature set to 22 degrees
Key Terms — Plain English
