The Undetected Damage of Quantization on Retrieval and How to Fix It
Luca Zhou, Alessandro Zirilli, Daniele Solombrino, Roberto Dessì, Emanuele Rodolà
cs.LG, cs.AI
2026-09-21
The same W4 checkpoint changes 3–9% of classification top-1s but 14–46% of retrieval top-1s. Gap-sensitive mixed precision recovers 60–73% of an extra bit at half the cost.
Post-training quantization is usually signed off on classification accuracy: if 4-bit barely moves the number, the job is done. Retrieval, RAG, recommendation, and reranking care about order. Hold the checkpoint and the quantizer fixed, switch only the task, and a model that keeps its classification top-1 still replaces a large share of retrieval top-1s.
Aggregate ranking metrics hide that. CLIP text-to-image Recall@1 falls 1.3%, yet about one in four top-1s change and one in ten previously correct queries lose their image. Qwen3-Embedding-8B loses 3.1% nDCG@10 while about one in six queries that had a relevant top hit lose it.
Treat quantization as a perturbation of at most ε on every score. The top-1 survives every such perturbation only if the gap g₂ to the runner-up is at least 2ε. Cross-entropy pushes the correct logit away from the rest; contrastive retrieval separates a positive from the paired negatives and never asks the winner to sit far from second place. The gap needs no labels. Before deployment it ranks which models will break. At serving time the quantized model's own top-1/top-2 gap is compared with a threshold calibrated on unlabeled data, which decides whether that answer still matches full precision.
Almost every retrieval query is at risk, so the fix is global: at a 3.5-bit average budget, spend the extra half-bit on layers whose solo quantization moves g₂ the most. Classification puts only a minority at risk, so the fix is per input: the quantized forward pass already produces the gap, and inputs below a threshold go to full precision.
The sweep covers ViT-B/16, ViT-L/16, Qwen3-Embedding 0.6B/4B/8B, plus GTE, BGE, E5, and CLIP, under RTN, GPTQ, AWQ, HQQ, and AdaRound, weight-only at W4 and W3.
Same backbone, W4, group128, round-to-nearest:
| Backbone | Classification top-1 change | Retrieval top-1 change | Ratio |
| ViT-B/16 | 6.2% | 45.5% | 7.3× |
| ViT-L/16 | 2.9% | 42.1% | 14.3× |
| Qwen3-Emb-0.6B | 4.3% | 33.4% | 7.8× |
Retrieval change at W4 runs about 14–46% depending on model and quantizer. CLIP ViT-L/14 on Flickr30k loses 1.3% Recall@1, changes 24.6% of top-1s, and drops the matching image for 10.9% of previously correct queries. On BEIR, Qwen3-Emb-8B's nDCG@10 falls 3.1% relative while gold leaves top-1 for 15.6% of queries that had it. Among flips whose full-precision top-1 was relevant, 66% (W4) and 77% (W3) land on a non-relevant document.
The gap splits the two families: 85.8% of classification inputs sit above the stability threshold against 3.5% of retrieval queries; median separation g₂/2ε is 5.73 versus 0.081. Across 2,118 setups the ratio tracks change rate at Spearman −0.88. Stronger quantizers help and do not finish the job: ViT-B/16 retrieval at W4 moves from 45.5% (RTN) to 30.6% (GPTQ with activation order), still far below the threshold. At W3, retrieval change often exceeds 50%.
Mixed precision: ranking layers by gap sensitivity at 3.5 bits recovers 60.2–73.4% of the W3→W4 reduction under every quantizer, ahead of reconstruction error (about 40–48%). On classification, routing the lowest-gap 25% of inputs to full precision recovers 85–93% of the accuracy lost to quantization, at about 47% of full-precision cost.
"Accuracy barely moved" is the wrong acceptance test for quantized retrievers. nDCG@10 is also blind to a top-1 swap when the old winner stays inside the window. For RAG, a different first document is a different generated answer, and two servers at different precisions disagree on the same query. The gap is label-free, so it can pick models before launch and refuse or reroute answers after. Retrieval should spend bits on layers that move the gap; classification should send the few risky inputs back to full precision. When choosing a retriever, separation beats size: GTE-large at 0.34B changes fewer top-1s at both bit-widths than Qwen3-Emb-4B.
Only weight-only 3/4-bit quantization; activations are untested. The stability check assumes calibration and traffic share a distribution. Retrieval is exact cosine over a finite corpus; ANN indexes add their own ranking error. Top-k stability is unused in practice because gaps below first place are another order of magnitude smaller. Classification routing sets the threshold at the 25th percentile of gq on a validation slice and must be recalibrated. The paper does not show that training to widen g₂ would make retrieval quantization-robust; that is left open.