Quantization silently scrambles 14-46% of top-1 retrieval results, label-free score-gap check offers a fix
_reachsumit · x · 2026-09-22
A new paper shows a quantized model can keep its classification accuracy while still changing 14-46% of its top-1 retrieval results — damage that aggregate ranking metrics only partially reveal.
Mechanism: The failure ties to the gap between the two highest scores. Classification losses push the correct class away from others, encouraging wide gaps; retrieval has nothing separating top-1 from second place. Top-1 is guaranteed to survive quantization only when the gap exceeds twice the largest rounding error, and this gap is measurable without labels.
Fixes:
- Pre-deployment: use the gap to predict which models will break under quantization.
- At deployment: decide per-input whether the quantized answer still matches full precision.
- Retrieval: spending extra bit-width on layers that move the gap most recovers up to three-quarters of an extra bit's benefit at half its cost.
- Classification: routing few low-gap inputs to full precision recovers most lost accuracy cheaply.
More from Infra
- Running 2.78T-param Kimi K3 on a single CPU in 8 GB RAM, no framework — tom_doerr · 2026-09-22
- LinkedIn's GPU Pre-Ranker Fuses Graph Features for 50x Bigger Models at 120ms p99 — _reachsumit · 2026-09-22
- oMLX Creator Jun Kim Joins Hugging Face to Bolster Apple's MLX Local AI Ecosystem — pcuenq · 2026-09-22
- Anthropic plans to scale to 5 GW of compute capacity by year-end — CurieuxExplorer · 2026-09-22
- Naveen Rao's new chip makes an AI image on ~500 nanojoules, thousands of times less than a GPU — victor_explore · 2026-09-22
- Sify Infinit Spaces IPO Hits Valuation Roadblock: Seeking Rs 40,000 Crore, Investors Say Rs 22,000-25,000 — HimanshiET · 2026-09-22