X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation
Haojun Zhang, Yi Zou, Min Chen, Qize Yu, Lianrui Fan, Xini Ding, Hao Li, Shuchang Zhou, Xianming Liu, Shiyu Huang
cs.SD, cs.AI
2026-09-10
X-AuT progressively prunes Qwen3-ASR-0.6B's audio tower from 18 to 16 layers with cross-scale distillation, cutting macro error on ten Chinese-English ASR sets from 5.61% to 5.27%.
In a speech LLM, the audio encoder runs over every frame. That cost shows up in first-token latency for streaming, mobile, and in-car systems. Dropping whole Transformer blocks keeps a regular architecture that is easy to ship. It also shifts the embeddings that the decoder conditions on, which can trigger early EOS and large deletions.
Most ASR compression either trains for variable depth, factorizes weights, or shrinks the decoder. XPeng starts from a finished Qwen3-ASR-0.6B checkpoint: an 18-layer audio tower with 186.4M parameters. Two questions are coupled. Which layers can be removed and recovered under a fixed budget, and how should recovery fix both hidden-state mismatch and errors that appear once the student conditions on its own prefixes. A layer that looks redundant alone can matter once its neighbor is gone, so static importance scores are not enough.
X-AuT filters transcripts, probes recoverable layer sets, then restores each hop in three stages. The language-model backbone stays frozen. Rank-32 LoRA on attention projections is trainable; the tied output embedding is trainable during distillation and frozen in the last stage.
The source pool exceeds 280k hours. Each reference-bearing clip is re-decoded by Qwen3-ASR-1.7B and Qwen3.5-Omni. Pairwise CER/WER among the source transcript and the two hypotheses produce nine consistency tiers. Reported runs use only class 1. Stage 2 keeps that filter and reweights sources toward cockpit queries and AISHELL-4/5.
Pruning is progressive: 18→16 drops original layers {1, 18}. For 16→14, every candidate starts from the same recovered 16-layer checkpoint and gets a matched 0.3-epoch LoRA warmup. Pair recovery is not additive. {6, 8} joins the two strongest single removals and still lands at 7.78% TER, 0.85 points behind the chosen adjacent pair {5, 6} at 6.93%.
The teacher is Qwen3-ASR-1.7B (24 audio layers, hidden width 2048 versus 1024). A two-layer MLP with a 256-d bottleneck maps teacher features into student space.
Two EOS guards sit on top: a trainable tied head and minnewtokens=3. Together they reach 6.75% dev TER and drop empty-rollout windows from 5 to 0.
Ten public Chinese-English sets, unweighted macro mean, single seed 42. No repeated runs.
| Model | Audio-tower params | Macro error |
| Qwen3-ASR-0.6B 18L | 186.4M | 5.61% |
| X-AuT 16L Stage 2 | 167.1M | 5.27% |
| X-AuT 14L Stage 2 | 147.8M | 5.75% |
| Self-distill 16L Stage 1 | 167.1M | 8.45% |
| 1.7B teacher 16L Stage 1 | 167.1M | 5.55% |
| Direct 18→14 | 147.8M | 6.73% |
The 16-layer model beats the baseline by 0.34 points (−6.1% relative). CommonVoice zh falls from 9.95% to 8.12%, CommonVoice en from 12.35% to 10.50%, WenetSpeech-meeting from 8.36% to 7.06%. Tedlium is the worst regression, +0.44. The 14-layer tower has 20.7% fewer parameters and 5.75% macro error, 0.14 above the baseline. Most of the extra loss sits on Fleurs-en (+0.93) and, versus 16 layers, CommonVoice en and the meeting set.
Teacher scale is the larger gap. Under a matched 16-layer recipe, the 1.7B teacher reaches 5.55% at Stage 1; self-distillation from the unpruned 0.6B model reaches 8.45% and loses on all ten sets. Direct pruning of the same four layers {1, 18, 5, 6} lands at 6.73% versus 5.75% for the two-hop path.
On an in-vehicle PPU, 14-layer encoder time drops from 14 ms to 11 ms (−21.4%); on H800, 88 ms to 78 ms (−11.4%). End-to-end gains are 4.7% and 2.6%, because autoregressive decoding dominates.
For speech LLMs that have to run on-device, this is a concrete recipe for shrinking a finished audio tower: probe recoverability, distill from a wider teacher, then mix in a little student-history supervision. Sixteen layers is the cleaner operating point and is more accurate than the original model. Fourteen layers trades about 0.14 macro points for 20.7% fewer audio-tower parameters.
Do not expect a large end-to-end speedup. Encoder pruning only trims the front of the pipeline.
All headline numbers are single runs. Checkpoint selection uses 25 utterances per development set, so a 0.14-point gap has no confidence interval. The study covers one model family and a small candidate set. The nine-tier pipeline is used only at class 1. Latency figures come from one retained benchmark dump. The self-teacher control also adds 2048→1024 projections, so the 8.45% versus 5.55% gap is not a pure teacher-ability effect.