Phoneme-Guided Initialization for LLM-based Speech Recognition
Ryo Magoshi, Shinsuke Sakai, Tatsuya Kawahara
eess.AS, cs.CL, cs.SD
2026-10-07
Pretrain the LLM on phoneme-to-grapheme conversion with text only, then fine-tune end to end. On 10-hour AISHELL-1, CER falls from 39.1% to 11.1% (72% relative).
A continuous speech LLM runs frames from an audio encoder through a projector into the LLM embedding space, then lets the LLM write the transcript. Closeness in that embedding space means similar meaning. Closeness in the encoder output means similar sound. With little paired speech and text, end-to-end training does not close the gap. Training only the projector is enough in some setups. Usually the LLM has to move as well, and LoRA does not remove the need for paired data. Text is much easier to get.
Cascades already win in the low-data regime. CTC predicts phonemes (speech-to-phoneme, S2P), then an LLM maps phonemes to text (phoneme-to-grapheme, P2G). Li and colleagues traced the end-to-end failure to the projector and showed that S2P initialization of the encoder helps. They did not train the LLM on P2G. A cascade also forwards its mistakes: one bad phoneme in spontaneous speech becomes a bad character.
Phonemes are a workable middle layer. The inventory is far smaller than Chinese characters or Japanese kana, so the encoder can learn it from fewer paired utterances. The LLM can learn from text alone which written form a phoneme string should take. Both sides are aimed at phonemes before joint fine-tuning.
The Kyoto University system uses XLS-R 300M as the encoder, a multilingual self-supervised model with its CNN front end frozen. The LLM is Qwen3.5-2B. Its image encoder and the paired projector are removed. The audio projector downsamples time by 4, then applies a two-layer MLP.
Without S2P initialization, end-to-end CER sits near 100%. Every end-to-end system below keeps that encoder initialization. Against Base, the proposed change is P2G initialization of the LLM.
Jie and Cheng, and Poncelet and colleagues, put phonemes in the decoded string, so the model has to emit them. Chinese can also interleave pinyin and characters, because each character is one syllable. Japanese kanji readings vary in length. Here phonemes are used only to set the initial weights.
Two kinds of scarcity. CSJ and AISHELL-1 are well-resourced corpora cut down on purpose: 10 hours and the 36-hour core of spontaneous Japanese, and 10 hours, 50 hours, and the full set of read Mandarin. Tatar and Urdu use all of Common Voice 25.0, 8.3 and 8.4 hours. Japanese and Chinese are scored with CER, the other two with WER. Base has S2P initialization and no P2G.
| Setup | Base | Base+P2G | XLS-R+CTC | Cascade |
| CSJ 10h CER | 20.8% | 11.7% | 13.8% | 18.1% |
| CSJ 36h CER | 13.5% | 9.8% | 10.7% | 14.8% |
| AISHELL-1 10h CER | 39.1% | 11.1% | 19.9% | 11.4% |
| AISHELL-1 50h CER | 11.2% | 5.7% | 10.1% | 8.1% |
| AISHELL-1 full CER | 6.5% | 4.1% | 6.5% | 7.0% |
| Tatar WER | 41.6% | 41.6% | 56.0% | 55.8% |
| Urdu WER | 29.3% | 28.4% | 30.0% | 28.8% |
On 10-hour CSJ, average CER moves from 20.8% to 11.7%, a 44% relative cut. The 36-hour core drops 27% relative. On AISHELL-1 the relative cuts are 72% at 10 hours (39.1% to 11.1%), 49% at 50 hours (11.2% to 5.7%), and 37% on the full set (6.5% to 4.1%). Less paired speech means a larger relative gain, and the full Chinese set still moves.
Without P2G, the 2B model loses to a 300M CTC baseline on 10-hour CSJ, 20.8% against 13.8%. With P2G it beats CTC on every CSJ set and beats the cascade on all four languages. The cascade loses where CTC phoneme errors propagate, which is spontaneous CSJ. At 10 hours of AISHELL-1 the cascade is still close, at 11.4%, because Mandarin phoneme-to-character mapping is fairly regular. Base+P2G reaches 11.1%.
A forced phoneme prefix depends on whether P2G was used. Without it, 10-hour Mandarin falls from 39.1% to 18.2%, while 10-hour Japanese rises from 20.8% to 24.3%. After P2G, phoneme-first decoding is worse than Base+P2G in every setting. The one edge is the joint model on 10-hour AISHELL-1 with phoneme-first decoding, at 11.0%, 0.1 points under Base+P2G.
Tatar is unchanged at 41.6%. The best Tatar run skips P2G and mixes the two output formats, at 40.6% WER. Urdu moves from 29.3% to 28.4%. Both scripts are more phonemic than characters or kanji: Tatar is Cyrillic, and Urdu writes consonants while omitting short vowels. Japanese phonemes in BCCWJ are manual, and Chinese uses g2pW, a tool built for polyphones. G2P for the other two languages is more generic.
The homophone share of substitutions is the slice where the sound was right and the written form was wrong. Across 15 Japanese and Mandarin comparisons, P2G raises that share every time. At 10 hours, Base writes 至关 as the unrelated 监管, and writes kinji 近似 as ninshiki 認識. Base+P2G writes 至 for 之, and swaps 回帰 with 快気, both kaiki. Adding paired speech pushes Base in the same direction. P2G gets there from text.
For Chinese and Japanese, text is cheap and in-domain paired speech is not. The LLM learns the sound-to-character map before it hears the target audio, and inference stays ordinary transcription.
The boundary is the writing system. A 72% relative CER cut at 10 hours of Mandarin and a zero on Tatar come from the same method. It fits little speech, plenty of text, and a script full of homophones. It does not fit every low-resource language.
On 10-hour CSJ, the 2B LLM is 7 points worse than CTC until P2G is added. After that, the same decoder beats both CTC and the cascade. Full AISHELL-1 still falls from 6.5% to 4.1%. The relative gain shrinks as paired data grows.
P2G initialization also shows the LLM extra target-language text: 2 million Chinese sentences and 1 million Japanese ones. The paper leaves the matched control for later, same text with ordinary language-model training and no phoneme-to-grapheme task. Errors move toward sound-faithful confusions, the same direction as adding paired speech, and Qwen already reads Chinese and Japanese, so the leftover looks like alignment, with limited room left for language-model knowledge. Until that control exists, how much of the CER drop is just next-token training is unknown.
Cutting hours in Japanese and Chinese is not a low-resource language. Those runs still have thick text, and Japanese phonemes are already in BCCWJ. Tatar has 65k text sentences and no gain. Languages that lack speech, text, and a usable G2P at the same time are outside these results.
Toneless pinyin counts tone pairs as homophones. 至 (zhì) and 之 (zhī) differ only in tone and still enter the share, so the number is high. A character lost to a tone error is untouched by this initialization.
The encoder still needs paired data. Without S2P, CER is near 100%. Skipping speech holds only for the LLM. Phonemes can be built from transcripts with G2P, but some speech-text pairs remain necessary. The only stack tested is XLS-R 300M with Qwen3.5-2B. English is absent. The cascade LLM is also adapted on the target transcripts, so its text exposure is not the same as the end-to-end systems.