Facto: factorized speech tokenizer yields stabler tokens for SpeechLLMs at NeurIPS
kastnerkyle · x · 2026-11-21
Researchers from Ubisoft La Forge, Concordia and Stellenbosch present Facto (NeurIPS 2026, arXiv:2611.18898): a speech tokenizer that first disentangles time-varying information (content + prosody) from static components (speaker identity, recording conditions) via a simple linear factorization, then clusters the factorized space.
Key points:
- Removing static components before clustering makes tokens markedly more robust to noise perturbations, validated across noise conditions;
- The paper provides theoretical convergence/scaling analysis showing factorization improves with the number of training speakers;
- A lightweight decoder reconstructs audio from the tokens;
- Greater token stability translates into improved text-to-speech performance and competitive downstream results, promising better SpeechLLMs.
More from Multimodal
- NVIDIA ships Alpamayo 2 Edge, a 2B-parameter in-vehicle reasoning VLA model — NVIDIAAI · 2026-11-21
- Creator shows HeyGen-powered video edit: 'editing to another level' — aziz4ai · 2026-11-21
- AI-built oil painting engine: 2,300 commits in 8 days as Claude Code runs a 'studio' of sessions — FotoRe_store · 2026-11-21
- "AGI Before GTA6": AI-Generated Music Video Riffs on 13-Year GTA VI Wait — Yamapama · 2026-11-21
- Meta's AVTR adds visual imagination to multimodal reasoning, +4.5 pts across 16 benchmarks — liuzhuang1234 · 2026-11-21
- Genie 4 world model live on Comfy playground, early testers impressed — EnvironmentalDay7419 · 2026-11-21