Whisper encoder pruned by 6 layers with label-free distillation, WER recovers to 20.1%
Rasmus Aagaard · hf · 2026-09-24
- Researchers propose pruning Whisper's encoder by ranking layers via leave-one-layer-out WER change, removing the 6 least impactful layers (18.5% of the encoder stack) — resulting in a shallower encoder that needs no custom inference code.
- To recover degradation from zero-shot pruning, they distill on unlabeled monolingual speech data: average WER across four languages drops from 21.9% to 20.1%, versus an 18.2% baseline.
- Context: decoder pruning is already widely adopted (whisper-large-v3-turbo cut the decoder from 32 to 4 layers; Distill-Whisper to 2), but encoder pruning lacked a practical approach until now.
- Code and the pruned model are open-sourced on GitHub and Hugging Face (whisper-large-v3-turbo-encoder-pruned).
More from Research
- GeoPair: Training-Free Cross-Layer Factorization Hits SOTA in Transformer Compression — MTSAIR · 2026-09-24
- Paper: Calibration Should Be a First-Class Criterion in LLM Evaluation — Mario Sanz-Guerrero · 2026-09-24
- FLEET: Entropy-Trajectory Memory Beats Repeated Sampling with 3x Speedup — Oleksii Streltsov · 2026-09-24
- CheatBench shows agents cheat: Kimi K3 at 72.3%, Grok 4.6 worst at 81.5% — davidmanheim · 2026-09-24
- Evidence on emergent misalignment is contradictory: values generalize but stay fragile — gleech · 2026-09-24
- Anthropic paper: verbalizable representations form a global workspace in LLMs — gleech · 2026-09-24