dRAE scales visual tokenization to 131,072 codes without codebook collapse
burny_tech · x · 2026-07-28
The paper introduces dRAE, a discrete representation autoencoder that uses hyper-spherical quantization instead of Euclidean codebook distances. The authors argue this avoids codebook collapse, better matches the geometry of visual representations, and allows much larger vocabularies.
According to the abstract, the method scales to 131,072 tokens, reaches 100% codebook utilization, simplifies training, and improves reconstruction, multimodal understanding, and image generation. The work positions angular routing as a better way to discretize high-dimensional visual features for language-model interfaces.
Related event: dRAE Scales Visual Discretization to 130k Tokens(2 posts)→
More from Multimodal
- 1,290-line commit lands MOSS Image support in ostris/ai-toolkit — ostrisai · 2026-10-16
- Ostris AI Toolkit adds training support for OpenMOSS's MOSS-Image-SFT model — ostrisai · 2026-10-16
- Nokia vs iPhone, car edition: AI image shows Tesla Roadster hovering over a Porsche — vaibhavbetter · 2026-10-16
- Midjourney multi-sref weight mixing recipe for cinematic light & motion portraits — michaelrabone · 2026-10-16
- MiniCPM-V 4.7 35B-A3B hits 17+ tokens/s on iPad with ~3GB memory — techNmak · 2026-10-16
- turbopixel: layer-and-mask generative compositing runs a 9B model on 4GB GPU — 3unjee · 2026-10-16