QUASAR fixes QAT's high/low-bit weight mismatch, cutting loss with zero inference overhead
arankomatsuzaki · x · 2026-10-17
This threads QUASAR, a new QAT method. QAT improves model quality at 2-4 bits but has a structural problem: each step applies gradient updates to high-precision latent weights while the loss and gradients are computed on their low-bit reconstructions—the weights actually used at inference.
This mismatch yields suboptimal training dynamics and a higher loss floor, which existing approaches (learned scales, soft-to-hard rounding, better init) don't address. QUASAR fixes it cheaply by only changing the weight reconstruction map in the forward pass, so it adds no inference overhead, composes with most other QAT methods, and consistently beats all QAT baselines on loss.
More from Models
- Vals RSI Index: Anthropic alone compressed RSI timelines by about a year, chart shows — thesaraharminta · 2026-10-17
- Reflection AI's Beam-1.0-501B-A23B debuts at #6 among US labs on Agent Arena — arena · 2026-10-17
- Reflection AI's open-source Beam-1.0-501B enters Agent Arena top 15 labs, ranked #6 among US labs — arena · 2026-10-17
- Musk: xAI building business-focused Hagglebot, touting up to $6T/year in global savings — elonmusk · 2026-10-17
- Microsoft MAI Base 2 and Meta Muse Spark Open Are the Week's Most Overlooked AI News — The AI Daily Brief · 2026-10-17
- User reports abnormal token consumption: 100% to 0% in a day plus 30k credits — MasterAyolos · 2026-10-17