Disclosure: I work on VoxRT (Elephant Enterprises LLC), the on-device voice runtime used as the example in this post.

Why compress voice models at all

If you work with LLMs, 60 MB sounds absurdly small for a model. In voice AI, it is the entry ticket. A streaming ASR model that runs real-time on a phone in the conformer/RNN-T family typically lands between 60 MB and 500 MB. A wake-word model that listens to the microphone 24/7 has to fit in roughly 100 KB with a sub-5% CPU budget. Three orders of magnitude between the two models in the same voice stack is not an anomaly, it is the design.

This article covers why voice models are large by default, which compression levers actually work for streaming audio, and two case studies from a production voice SDK: a streaming ASR model at 60 MB and a wake-word model at 100 KB. All numbers are verifiable against release artifacts in our public repositories.

Why voice models are large by default

The default size comes from three places.

Attention windows run wider than most people expect. Streaming ASR needs roughly 1 to 2 seconds of left-context (cached from prior chunks) plus a shorter right-context window per chunk for decent accuracy, not per-frame prediction. Transformer-class or conformer-class models with internal attention over 100-plus frames end up multiplicatively larger than an event detector that only sees 300 to 500 ms.

Mel-feature extraction and audio preprocessing are tensor-heavy. Audio arrives at 16,000 samples per second. To process it efficiently, models use a convolutional stem (depth-wise separable or regular) that compresses the temporal dimension down to a frame rate the ML model works at natively (typically 50 to 100 Hz). That stem is typically under 1 MB in modern FastConformer-style subsampling, several MB in older designs.

Vocabulary tables. ASR models need embedding tables for subword tokens. At typical embedding dimensions this is roughly 1 to 5 MB per table, before any model weights are counted. LLMs face this at tens of MB scale, ASR less so, but it is non-zero.

Add it up for a production streaming ASR model and you land in the 500 MB to 2 GB range uncompressed, with non-streaming offline models like Whisper large-v3 reaching ~3.1 GB. For mobile deployment, that is too much.

The compression toolkit

Quantization. INT8 inference weights instead of FP32 training weights is the mobile ML standard. Memory footprint drops roughly 4x, latency on well-tuned INT8 kernels is comparable to FP16. For streaming ASR, attention paths often need an FP16 fallback. Streaming decoders carry predictor state across chunks, which exposes two issues you do not hit in offline inference: predictor state drift under low-bit quantization, and activation range mismatch at chunk boundaries where calibration from one window does not match the next. What works for Whisper-class batch inference does not transfer directly. Feature extraction and convolutional stems quantize to INT8 cleanly either way.

Architectural choices at design time. Depth-wise separable convolutions instead of regular ones (used in MobileNet-family work, directly applicable to audio stems). Streaming attention via chunked or cache-aware patterns, where the model attends over a finite window instead of full-sequence attention. These choices are locked in model architecture, not post-training compression. They are the initial trade-off.

Vocabulary tightening. For a wake-word model the vocabulary is one class (phrase present or not). For keyword spotting it is a fixed closed set (14 commands in our case). For open-vocabulary ASR you need a subword vocabulary of roughly 5,000 to 10,000 tokens. The tighter the vocabulary, the smaller the output layer, the smaller the model.

Knowledge distillation. Teacher-student training where a large model teaches a small one. Works well for non-streaming ASR, needs more care for streaming architectures (the student has to carry its own streaming receptive field). Not a universal fix.

These four levers combine in practice. Production voice models usually use two or three at once, rarely a single technique.

Case study 1: streaming ASR at 60 MB

Our streaming ASR SDK (voxrt-asr-linux, voxrt-asr-ios, voxrt-asr-android) is based on the NeMo FastConformer family. The inference artifact is a single .vxrt file at 60.4 MB per the v0.1.2 release (verifiable at github.com/VoxRT/voxrt-asr-models/releases).

What this means in practice:

• The model fits in memory on any current mobile device without OOM concerns
• Real-time streaming inference is achievable on a mid-range Android (Snapdragon 662-class)
• The model file fits in a single APK bundle without needing split APKs or dynamic downloads for the typical case

What it means trade-off-wise:

• Open-vocabulary. Works with general English speech, not a closed set
• Streaming native. Emits partial transcripts incrementally (chunk size is 200 ms, 3200 samples at 16 kHz)
• English-focused. Multilingual support is not shipping in this SDK yet
• RNN-T decoding path. Latency versus accuracy balance tuned for voice assistant and dictation use cases

Rough comparison with cloud ASR models for perspective:

• Whisper large-v3 (OpenAI, offline): about 3.1 GB
• Google Chirp (cloud): internal, likely multi-GB
• Our streaming model: 60 MB

Roughly 50x smaller than Whisper large. Not free. The accuracy trade-off is real. On WER tests over LibriSpeech test-other, Whisper large-v3 is better. The comparison is also somewhat lopsided, because Whisper is a non-streaming offline model while our ASR targets streaming use. Streaming RNN-T and offline sequence-to-sequence live in different categories. The engineering trade-off that drives mobile voice input is streaming behavior plus battery, latency on the first partial, and memory footprint that leaves room for the rest of your app. On those axes a 60 MB streaming model is the right fit for mobile voice input, even when WER is slightly higher on offline benchmarks.

Case study 2: wake-word at 100 KB

The other end of the spectrum. Our wake-word model (voxrt-wake-word-linux, voxrt-wake-word-ios, voxrt-wake-word-android) ships as a 100 KB model file with a single trigger phrase, “Hey Assistant”. Measured RTF is 5.3% of one CPU core, sustained, on one A53 core of a Raspberry Pi Zero 2 W ($15 hardware, 1 GHz, 32-bit Raspberry Pi OS).

What makes a 100 KB target reachable:

• Event detector architecture, not sequence-to-sequence
• Single-class classification (phrase present or not), not multi-class
• Short receptive field (enough for 300 to 500 ms phrase detection)
• No vocabulary table, just one trained trigger
• Architectural choices tuned for tiny CPU footprint

Trade-offs accepted to hit 100 KB:

• Fixed trigger phrase. “Hey Assistant” only. Custom per-user triggers would require a separate training pipeline, which is not a public SDK feature today
• English speech patterns. The model is trained on English, performs less well on accented English or other languages
• False accept vs false reject tuning. The default threshold is 0.9, settable. Lower threshold means more wakes but more false triggers from TV or music

Why this matters engineering-wise. A wake-word runs 24/7 if you want an “always listening” UX. Even a 10% CPU budget sustained means noticeable battery drain. 5% is the comfortable sweet spot. It leaves battery for the rest of your app, allows thermal throttling margin, and fits on the cheapest edge hardware you can buy.

Trade-offs we accepted and why

The explicit constraints of our model lineup:

Fixed vocabulary for event detectors. Wake-word is one phrase, KWS is 14 fixed commands. Custom vocabulary requires a training pipeline we have not released as a public SDK feature. We ship well-tuned fixed vocabulary now, with custom training as a later priority. Users who want custom triggers have options (train upstream FastConformer themselves, contract for custom work).

English-focused models. Multilingual on-device models are large and accuracy drops versus English-only. We chose to ship English well first and multilingual later. Honest claim: our models are not a replacement for native-language voice models that already exist for specific languages.

Model updates through the SDK release cycle. Cloud ASR teams push improved models to the server and tomorrow every user has better accuracy. For us, that means a new SDK version, npm/pip/CocoaPods updates, and users having to upgrade. Standard mobile library pain. A stable C ABI minimizes breakage but does not remove it.

Where compression breaks down

Honest limits where these trade-offs start to hurt:

Multilingual coverage. Open multilingual ASR models (Whisper multilingual, Google USM) are still noticeably capable for low-resource languages. A compressed English-focused model is not an alternative if your user base regularly speaks 50-plus languages. For English-dominant products, fine. For truly global products, a real limitation.

Low-SNR acoustic environments. Compression trade-offs bite harder in noisy conditions. A wake-word running against a loud TV or music in the background performs noticeably worse than a larger cloud wake-word. Not fatal, but visible if you benchmark honestly across conditions.

Speaker invariance. We have observed occasional speaker-specific failures that appear tied to attention quantization and parameter-sharing shortcuts. Training on a broad speaker set minimizes it, but compressed models sometimes miss specific voice characteristics (deeper voices, strong accents, speech impediments) that full-precision models catch.

Aggressive sub-100 KB targets. Below 100 KB for a wake-word, we hit diminishing returns for our architecture. Porcupine and some Google Hotword models ship smaller, so this is not a universal boundary. 100 KB is our current sweet spot given our architecture choice and the $15 Pi hardware we target. Smarter engineering could push it lower, but the ROI is narrow for our use case.

Closing thoughts

Not every voice task needs an LLM-sized model. A 60 MB streaming ASR and a 100 KB wake-word are not compromise versions of bigger models. They are purpose-built for specific hardware constraints. The trade-offs are real, and we do not hide them:

• Vocabulary limits
• Multilingual coverage is weaker
• Updates ride the SDK release cycle
• Specific acoustic conditions where compression hurts

For mobile voice input, edge voice control, and IoT voice interfaces, these trade-offs are acceptable. For production cloud transcription with full language coverage and server-grade latency tolerance, a different trade-off wins.

What is worth watching in 2026–2027: new silicon on mobile devices (Apple Neural Engine, Qualcomm Hexagon NPU, Google Tensor TPU) shifts what is practical on-device. A model that needed 100 MB to run real-time in 2024 could plausibly land at 60 MB with comparable quality using newer streaming conformer architectures. The compression toolkit keeps evolving, and released-artifact numbers beat aspirational benchmarks for tracking where things actually land.

All technical claims (model sizes, benchmarks, hardware specifications) were verified against VoxRT release artifacts (voxrt-asr-models, voxrt-wake-word-linux) and public NeMo documentation before publication. More on the runtime at voxrt.com.

Model compression trade-offs for streaming voice AI was originally published in Neural Notions on Medium, where people are continuing the conversation by highlighting and responding to this story.

Add four short lines to your prompts and Claude marks the facts it doesn’t have, labels its guesses and lists what it cut or added. Without them, it can fill a gap with something that sounds right, like a texture or a claim about past clients.

This article was drafted with help from Claude, the tool it’s about. Every prompt in it was tested before publishing.

The runs use Maya, a fictional copywriter for small skincare brands. Her Claude project holds her instructions and a knowledge file with her rates, fictional past clients and work samples. Each prompt ran in a new chat inside that project on claude.ai in October 2026, on a paid plan (Team), with Claude Opus 5.5. Runs are named by their prompt ID in the Claude for Freelancers book. Each [trimmed] line marks a cut. Claude’s output varies between runs, so yours will differ.

1. Mark every missing fact

Never invent results, clients or numbers. If a draft needs a fact you don't have, write NOT IN NOTES.

This line is in Maya’s project instructions, so it applies to every chat in her project. A marker in capitals is easy to search for before you send. It turns a missing fact into a question for you instead of a guess in the client’s inbox.

C2‑05 asked for a pitch email to Larchwell, a fictional skincare brand, with one small idea, such as a few lines for a product page. Maya’s notes say: “Product pages are one sentence plus the ingredient list. Nothing on texture, scent or how to use.” The prompt repeats the rule and names the details it expects: “such as texture, scent, how to use it or a date”.

Claude’s output, C2‑05, October 12, 2026:

[trimmed]
EXAMPLE ONLY
 
Vitamin C Serum, $38
 
Vitamin C, plus NOT IN NOTES (rest of the ingredients, in plain words). That's most of it.
 
Texture: NOT IN NOTES.
Scent: NOT IN NOTES.
How to use: NOT IN NOTES (morning or night, how many drops).
Not for: NOT IN NOTES.
[trimmed]

The sample is mostly gaps, and each gap matches something the notes leave out. Fill them from the client’s own product information, or cut the lines.

2. Ask for the quote and a Fact or Guess label

The line from my notes it rests on, quoted.
"Fact" if my notes state it, "Guess" if it depends on anything they don't say.

These are two of the three things C2‑01 asks for under each reason to contact a brand now. The quote lets you check a point against your notes in seconds. The label separates what you wrote from Claude’s reading of it.

In this run, the notes on Larchwell said “Priya is the founder. Started 2023, small lab since 2024.” Maya’s project files say her best clients have 1 to 10 people.

Claude’s output, C2‑01, October 12, 2026:

[trimmed]
4. It's a small, founder-led brand, the kind you work with best.
Quote: "Priya is the founder. Started 2023, small lab since 2024."
Guess. Founder-led is stated. Team size isn't, and "small lab" doesn't confirm 1 to 10 people.
 
5. A launch with no email list is a gap a launch sequence could help with.
Quote: "No email signup on the site that I could find."
Guess. They may collect emails somewhere you didn't see, and your notes don't say they want email at all.
[trimmed]

Each reason reads like a fact until its label. Claude then kept email out of the suggested angle: “Save email for a later conversation, since reason 5 is a guess.”

3. Shorten without changing the meaning

Don't add anything that isn't in my draft.
 
Give me the short version first. Then list every cut, one line each, so I can put back anything that mattered.

A shorter draft can lose a claim along with the extra words. The first line keeps new material out. The list shows every change, so you can undo the ones that mattered.

C2‑07 cut Maya’s first draft of a pitch to Larchwell to under 100 words. Her draft said she writes for “skincare and beauty brands” and has “worked with many skincare brands”. The run listed 15 cuts.

Claude’s output, C2‑07, October 12, 2026:

[trimmed]
Cuts:
[trimmed]
- "and beauty" was cut because your notes say skincare only.
[trimmed]
- "I have worked with many skincare brands" was cut because "many" isn't backed by your notes.
[trimmed]

Both cuts rest on Maya’s project files, which say she works in skincare only and name three past clients. They are good cuts, and they still change what the pitch claims, so read them before you accept the short version. The short version adds no fact that isn’t in the draft.

4. Make Claude list what it added

After the summary, list every line you wrote that isn't in my notes, so I can check it.

Claude fills small gaps so a draft reads smoothly. This line makes it point to each one.

C3‑02 turned Maya’s call notes into a scope summary for Leon and Ines, who run Tidewick, a fictional seaweed balm brand, to confirm in one reply. The prompt also said “Use only my call notes, not my project files or terms.”

Claude’s output, C3‑02, October 12, 2026:

[trimmed]
**Lines not in your notes:**
[trimmed]
- "Yours are ready Mon Nov 2" under Not included. Your notes say photos are ready, but not who supplies them. I assumed Leon and Ines do.
- "Uploading the pages to Shopify." Your notes only say the shop runs on Shopify. They don't say who uploads.
[trimmed]
- "launch week starts." Your notes say "launch week Nov 23" but don't say the week starts on that date.
[trimmed]

The first two are guesses about who does what. Once the client replies “confirmed”, guesses like these are part of the agreed scope, so settle each one first. The same summary wrote NOT IN NOTES for revision rounds and for the first draft date, which the notes don’t mention.

What these lines don’t do

They don’t check facts for you. A quote can be copied wrong, and a Fact label only means your notes say it. Claude can also miss an addition or a cut in its own list. Before anything reaches a client, find each quote in the source and fill in or cut every NOT IN NOTES. Read each list of cuts or additions in full.

Kurows is not affiliated with Anthropic. Kurows sells the Claude for Freelancers book these runs come from, and the link below goes to its free Mini.

The prompts behind the first three runs, C2‑05, C2‑01 and C2‑07, are among the 10 tested cold pitch prompts in the free Claude for Freelancers Mini: get the Mini here.

4 Guardrail Lines That Make Claude Flag Missing Facts Instead of Inventing Them was originally published in Neural Notions on Medium, where people are continuing the conversation by highlighting and responding to this story.