What Does Quantization Actually Lose? Testing BF16 to Q2 Word-by-Word with Qwen 27B
TheMoonMidas · x · 2026-09-09
The original thread (@nettermina) asks what the "nearly" in "nearly lossless" quantization actually means: models are trained in BF16, released in Q8, and commonly run at Q4 — but what gets lost between BF16 and Q2?
The author ran a visualization experiment: had Qwen 3 32B (BF16) write a paragraph, then had every quant level below it generate the same content, marking the mutations word by word. Key finding: every token a model emits carries a confidence score. Words the BF16 model is 90% sure about rarely change even down to Q2, while words at only 40–50% confidence (those with multiple plausible alternatives, like windy/sunny/cloudy) are exactly what quantization degrades.
More from Research
- WaveNet turns 10: audio-as-language-modeling is now the standard, dilated convs still everywhere — heiga_zen · 2026-09-09
- NYU prof Tristan Buckmaster offers alternative view on OpenAI's Navier-Stokes claim — JitendraMalikCV · 2026-09-09
- Why Matrix Multiplication Keeps Showing Up in Unrelated Math Fields — burny_tech · 2026-09-09
- Closed-form Jensen-Shannon divergences via abstract means: an arXiv generalization — FrnkNlsn · 2026-09-09
- AI agents learn to "time-travel" and sacrifice themselves to help the swarm — thlarsen · 2026-09-09
- Azure PII removal tool fails to protect 74% of info in MedQA, privacy paper finds — niloofar_mire · 2026-09-09