MPEcho: A Melody and Phoneme-Aware Generative Framework for Controllable Cover Song Generation
Wei-Jaw Lee, Hsuan-Yu Yeh, Ting-Yi Hu, Chih-Pin Tan, Fang-Duo Tsai, Yi-Hsuan Yang
cs.SD, cs.AI, eess.AS
2026-07-29
Adding a phoneme encoder and length regulator to SongEcho, plus the Whisper-based singing annotator Phonsa, cuts phoneme error rate from 45.62% to 18.65% with melody intact.
Cover song generation (CSG) keeps a reference song's melody and lyrics and regenerates the rest (accompaniment, timbre, arrangement). The melody side is already handled well by SongEcho, the current state of the art. The hard part is the lyrics. SongEcho conditions on a pitch contour (F0) and a voiced/unvoiced (V/UV) flag. V/UV only marks whether a frame contains a human voice; it carries no phoneme information, so the model has to guess which word is sung from context alone. The result is garbled lyrics, with a phoneme error rate (PER) of 45.62%. Nearly half the phonemes are wrong. A cover that nails the melody but mangles the words is not much use.
MPEcho borrows from singing voice synthesis (SVS), the field that has always modeled lyrics at the phoneme level. It adds two components to SongEcho:
The phoneme timestamps have to come from somewhere, and high-quality audio-to-phoneme pairs are scarce. So the authors built a second contribution, Phonsa, a Whisper-based singing annotator. Compared with prior work it drops from word-level to phoneme-level modeling, replaces the RNN with chunked self-attention (500-frame, 10-second chunks, 50% overlap), and adds breath and boundary tokens. It trains on M4Singer and Opencpop with a CTC plus cross-entropy multi-task loss.
Phoneme arrangement matters. The authors compare three styles: word-level, a JAM style with filler tokens, and their SVS style with phoneme-level timestamps plus LR. The JAM style backfires, pushing PER to 0.7125 because the filler tokens confuse the model. The SVS style is the one that works.
At inference they use multi-condition classifier-free guidance, weighting text, lyrics, and time-varying controls (melody plus phoneme) separately, with adaptive projection guidance.
Phonsa alignment quality (Table 1):
| Model | Alignment MAE |
| MFA (baseline) | 233.9 ms |
| Phonsa | 32.6 ms |
Alignment error drops from 233.9 ms to 32.6 ms, about one-seventh of the baseline.
Main CSG results (Table 2; lower PER is better):
| Model | Condition | PER | Melody RPA |
| SongEcho | melody only | 0.4562 | 0.5779 |
| MPEcho | phoneme only | 0.2292 | 0.0667 |
| MPEcho | melody + phoneme | 0.1865 | 0.5764 |
| ACE-Step baseline | n/a | 0.4348 | 0.2581 |
With phoneme conditioning, PER falls from 0.4562 to 0.1865, a relative reduction of about 59%, while melody consistency (RPA 0.5764 vs 0.5779) barely moves. Phoneme alone also gets a low PER (0.2292) but collapses the melody; both conditions are needed. The only reference system, ACE-Step, scores 0.4348 PER.
In the subjective test (Table 4, MOS, 25 listeners), the melody-plus-phoneme configuration with multi-condition guidance scores highest on all metrics: overall 3.57, melody consistency 3.88, vocal naturalness 3.59.
This shows that bringing mature SVS phoneme alignment into a modern diffusion-based cover-song model directly fixes the wrong-lyrics problem without sacrificing melody. Phonsa is a useful by-product for any task that needs singing phoneme alignment. For anyone building karaoke, music generation, or cover-song tools, it is a ready improvement path. It is a solid engineering advance, not a paradigm shift.
The authors concede MPEcho only handles single-singer scenarios; multi-singer, multilingual, and richer prosodic control are left to future work. Boundary detection remains hard, with a boundary F1 of only 0.534 under a strict 20 ms tolerance. CLAP scores are low for every method, which the authors attribute to a domain mismatch between CLAP's training data and their Chinese old-song distribution. That suggests standard audio-text metrics may not fairly evaluate such models, so the objective conclusions deserve a discount. The paper offers no broad head-to-head with mainstream singing or cover models; ACE-Step is the only reference point.