Pruned CTC for Memory-Efficient Large-Vocabulary ASR Training
Yifan Yang, Xiaoyu Yang, Zengrui Jin, Xian Shi, Yuxuan Wang, Yu Xi, Ziyang Ma, Qi Chen, Ruiyang Xu, Hui Wang, Dongchao Yang, Jin Xu, Xie Chen
eess.AS
2026-09-27
Pruned CTC matches full CTC while aligning only batch targets plus blank. At 180K vocab it cuts step memory 5.1x; LLM-CTC stays within 7% relative WER of CE and is 7 to 10x faster.
LLM-based ASR usually trains with token-level cross-entropy and autoregressive decoding (LLM-CE). Generation is sequential at train and test time. Streaming adds another tax: each audio chunk needs a matching text span, often from an external aligner.
CTC (Connectionist Temporal Classification: align variable-length frames to a transcript by summing paths that collapse to it) can train from an utterance-level transcript and decode offline or streaming. Standard code still materializes a frame-by-vocabulary activation tensor. Native LLM vocabularies run to 150K–180K classes. At that width the logit tensor can dwarf encoder activations and blow the memory budget.
Every valid CTC path uses only blank (the empty class that occupies a frame, then disappears under collapse) and the tokens in that utterance. Across a batch, their union is typically a thin slice of V. The remaining columns exist so the softmax normalizer is correct. The objective was fine. The layout was not.
Pruned CTC splits alignment from normalization. The dynamic program runs only on blank plus the distinct target tokens in the batch. Softmax still sums over the full vocabulary, so dropped classes keep contributing through the normalizer and still receive dense head gradients.
The lattice searches only the stations this sentence actually uses, while each step still divides by the full-line sum. In exact arithmetic the reduced computation matches full-vocabulary CTC in both loss and first-order gradients. Classes outside the utterance have zero occupancy (frame-level posterior given the transcript); their gradient is just their softmax probability.
Two implementation choices keep the M-by-V tensor off the resident set:
LLM-CTC hangs a pretrained causal LLM on this loss, keeping causal attention and the native vocabulary. The SPEAR-XLarge v2 speech encoder and the LLM head stay frozen; only the projector and LoRA train. Speech-readout scores CTC at speech positions, which see a growing prefix. Query-readout appends Tn copies of one learned vector after the speech embeddings, so every CTC position can attend to the whole utterance under causal attention. The paper uses query-readout because that same causal mask streams: a query sees the current chunk plus bounded history. Blank sits on an unused tokenizer row, with a learned scalar offset η.
Streaming cuts audio into 2-second chunks. Encoder and LLM share a bounded left history. CTC is still summed over the full transcript, so training never needs chunk-level alignments. At inference the LLM reuses a KV cache and CTC prefix beam search continues across chunks.
On Zipformer-M and one H100 80G, Pruned CTC head-and-loss memory stays roughly flat from 10K to 180K classes. At M=16,000 and V=180,000 it falls from 43.0 GiB to 1.29 GiB (33.3x). Full training steps, at the largest vocabulary where standard CTC still finishes:
| Setting | Standard CTC | Pruned CTC | Memory | Step time |
| M=16k, V=180k | 51.7 GiB | 10.1 GiB | 5.1x | +17% |
| M=32k, V=130k | 77.3 GiB | 17.9 GiB | 4.3x | +17% |
| M=64k, V=50k | 76.2 GiB | 33.8 GiB | 2.3x | +14% |
At V=180k with M=32k or 64k, standard CTC runs out of memory. Pruned CTC finishes those steps at 18.2 and 34.5 GiB.
WER and CER match standard CTC to two decimals on LibriSpeech, GigaSpeech and AISHELL-1: 2.52 / 2.52 on test-clean, 6.02 / 6.02 on test-other, 11.27 vs 11.24 on GigaSpeech test, 4.80 vs 4.79 on AISHELL-1 test.
On GigaSpeech, SPEAR-Qwen3 from 0.6B to 32B keeps LLM-CTC within 7% relative test WER of LLM-CE, with 7.2 to 10.3x better RTF (recognition time over audio duration). At 32B, CE is 9.41 WER at RTF 0.1546; CTC is 9.94 at 0.0150. From 4B up, LLM-CTC also beats the same encoder's SPEAR-XLarge CTC (10.41) and RNN-T (10.26) on test. On one A100, 8B full-step memory drops from 56.06 to 34.67 GiB (38.2%) at +7.5% step time; head-and-loss falls from 29.21 to 1.14 GiB (25.7x). Unpruned reduced CTC already matches those full-step numbers, so vocabulary reduction plus recomputation do most of the work.
A 4B readout ablation: query-readout under causal attention is 10.21 test WER versus 10.46 for speech-readout; full bidirectional speech-readout reaches 10.16 but cannot stream with a KV cache. Fine-tuning Qwen3-ASR 0.6B and 1.7B for streaming raises test WER by less than 3% relative to matched offline models (0.6B: 10.73 offline vs 10.98 with 2s history; 1.7B: 10.16 vs 10.35). Stretching history from 2s to 8s barely moves WER.
Cut Cross-Entropy already showed that token-level CE need not store a full-vocab logit tensor. CTC has no frame targets, so occupancy comes from forward-backward over the transcript. The move that makes the same trick valid is separating the alignment support from full-vocabulary normalization. For Zipformer plus k2 recipes, swapping the loss is enough to train huge vocabularies without giving up accuracy on three corpora.
For LLM ASR this is a workable non-autoregressive adaptation: causal attention stays, the native vocabulary stays, and streaming does not need chunk-level alignments. A 7% relative WER gap in exchange for 7 to 10x faster recognition is a real trade for latency-sensitive offline decoding. No new acoustic model here. CTC becomes trainable at native LLM width.
Full-step memory on LLMs only falls by about a third, because weights and other activations still dominate. The 25x saving sits in the head and loss.
There is no dedicated Limitations section. Finite-beam pruning overestimates the full alignment sum and can make the loss discontinuous at prune boundaries. β=100 drives discarded posterior mass to 10^{-11} at random initialization; that is not a certificate over a full training run.
When the two gradient terms nearly cancel on selected classes, rounding takes over. The appendix reports relative weight-gradient error up to 1.21×10^{-4}, mainly from FP32 accumulation. Streaming training scores all chunks in parallel; inference walks them with a KV cache. The two agree in exact arithmetic. Under BF16 plus FlashAttention 2, relative logit error on Qwen3-ASR 0.6B is about 0.054 to 0.057, and greedy frame labels disagree. Only FP32 unfused attention is essentially exact. A train/serve precision mismatch is not locked down.
LLM-CTC freezes the encoder and the head. How much of the 7% CE gap is CTC's frame-independence assumption versus the freeze is unmeasured. LLM numbers are almost entirely English GigaSpeech. Longer left history barely helps, and the paper does not diagnose whether bounded context is actually used.