Aurora2-400M-A20M-QAT: a 0.5-bit MoE checkpoint shrunk from 799 MB to 27.5 MB

Tall_Abrocoma_3533 · reddit · 2026-11-23

The author releases a quantization-aware-trained version of Aurora2-400M-A20M using LittleBit-2 binary low-rank quantization with distillation from the original model. The routed experts (98% of parameters) are stored at roughly 0.5 bits per weight, shrinking the checkpoint from 799 MB in FP16 to 27.5 MB.

The quality cost is large: the Intelligence Index drops from 21.38 to 10.83. Arithmetic holds, PIQA stays above chance, while HellaSwag and ARC-Challenge fall near chance.

The compression is storage-only: without dedicated binary kernels, the loader expands weights back to FP32, so memory and speed are no better than the base model. It targets research on extreme low-bit MoE quantization, not a drop-in replacement.

Original post →

More from Models

Models channel →