StableVQ: three lightweight fixes for stable vector-quantized tokenizer training
Kwai-Kolors · hf · 2026-09-23
Kwai-Kolors (Kuaishou) released StableVQ, a systematic take on the underexplored problem of training stability in discrete visual tokenizers.
Diagnosis
- VQ powers modern autoregressive and masked image generation models; shared-projection codebooks improved utilization, but training remains fragile.
- The authors trace the root cause to entanglement of Encoder–Decoder and Codebook training: neither module fulfills its role independently, so the system only works when the two happen to cooperate — breaking down exactly when training is most stressed.
Three fixes
- Dynamic STE: corrects instability in the encoder's objective, enabling robust reconstruction optimization even with low codebook utilization.
- Region VQ Loss: reconceives the codebook objective so it independently guarantees full tracking of the encoder output distribution.
- Decoupled Schedule: assigns each subsystem its own learning-rate schedule to match their distinct responsibilities.
Results: Built on shared-projection codebooks, StableVQ is lightweight and adds no learnable parameters; on ImageNet it consistently improves training stability, codebook utilization, and reconstruction quality across codebook sizes and initializations.
More from Multimodal
- LOTR 'Fellowship of the Muscles': AI Video Turns the Fellowship Into Bodybuilders — iquizuanswer · 2026-09-23
- AI-edited soldering video auto-chapters footage and fixes verbal slips — debreuil · 2026-09-23
- Comfy Desktop ships early-stage performance testing for local ComfyUI instances — Lexius2129 · 2026-09-23
- MiniMax H3 powers retro-style short film honoring Earth, Wind & Fire — TheChuckTone · 2026-09-23
- Swarm-built inference engine runs Qwen Image-2.1: 1K images in under 0.5s — bingxu_ · 2026-09-23
- Gemini image generation stops refining and regenerates whole scenes on follow-up edits — asm99 · 2026-09-23