dRAE uses hyperspherical codes to scale visual discretization to 131,072 entries
antgroup · hf · 2026-07-28
This work tackles how to discretize high-dimensional visual representations so they can bridge better to language models.
- The authors argue that common Euclidean codebook objectives suffer from a metric mismatch with anisotropic representation geometry.
- That mismatch leads to unstable magnitude scales, uneven angular distributions, and codebook collapse.
- They propose Hyper-Spherical Quantization (HSQ), which separates semantic content from feature magnitude via angular routing.
- The resulting discrete autoencoder, dRAE, is designed to preserve semantic integrity while scaling the codebook.
- In experiments, the vocabulary scales to 131,072 entries with 100% codebook utilization.
- The report also claims strong reconstruction and downstream gains across understanding and generation tasks.
Related event: dRAE Scales Visual Discretization to 130k Tokens(2 posts)→
More from Multimodal
- Elorian launches Glimpse, an Astra-level visual reasoning model at 17x lower cost — CurieuxExplorer · 2026-10-17
- MOSS-Image base model showcase: cfg4, 39 steps, ~1.5MP renders — COMPLOGICGADH · 2026-10-17
- MOSS image model 4K test: 55s per frame on RTX 5090, 19.5GB peak VRAM — RauschCraft · 2026-10-17
- Fizgig LoRA Isolation: 0/50 face bleed-through vs 48/50, likeness kept — shootthesound · 2026-10-17
- MiniCPM-V 4.7 runs a 35B multimodal model in 3GB of memory, fully on-device — alifcoder · 2026-10-17
- Open-source .char format now supports up to 4 consistent characters per generation — ashishsanu · 2026-10-17