Ternary weights shrink a speech model from 2,472 MB to 181 MB with just +0.4pt WER
lmoroney · x · 2026-10-25
Laurence Moroney shares a hands-on extreme quantization case: constraining 98% of NVIDIA Parakeet-TDT-0.6B-v2's encoder parameters to just -1, 0, or +1 (one scale per row) cuts the weight file from 2,472 MB to 180.8 MB, while mean WER across seven test sets rises only from 6.45% to 6.84%.
Accuracy is largely recovered via quantization-aware training with distillation from the original model: 250,000 steps on 33,000 hours of audio, taking 69.3 hours on a single RTX 5090. The model card is candid about weak spots, with the biggest gap on Common Voice, which wasn't in the training mix.
His learning tip: to really understand quantization, read a model card like this end to end and try to reproduce one row of its results table.
More from Models
- Apple's on-device Foundation Model tested: 45/49 on routing, but easily prompt-injected — NellWatson · 2026-10-25
- Blind test of 5 local AI agents on one Qwen model: scores range from 3/5 to 5/5 — GapNew4766 · 2026-10-25
- DLSS 5 works stunningly on 25-year-old FFXI despite almost no training data — eschadiol · 2026-10-25
- AAI Index #4 Kimi 3.1 Pro badly underperforms MiMo 2.6 and DeepSeek 4.1 Flash in real use — PerformanceRound7913 · 2026-10-25
- Kimi K3.1 Pro tops offensive cyber eval at 46.43%, claims of deliberate nerfing surface — teortaxesTex · 2026-10-25
- OpenAI and Anthropic reportedly blocked REA tool reverse engineering privacy-invading smart TVs — kroggens · 2026-10-25