Qdrant: collection outgrowing RAM pushes rescoring latency from 4.1ms to 43.4ms
qdrant_engine · x · 2026-11-02
Qdrant's engineering blog quantifies the cost of exceeding RAM: in a 4.6M-vector test, rescoring latency jumped from 4.1ms to 43.4ms when RAM dropped from 12 to 4 GiB — the culprit being disk reads of full-precision originals.
Key takeaways:
- Quantization keeps a compressed copy in RAM while full-precision originals live on disk; TurboQuant rotates vectors before compressing (bits4 down to bits1), with bits4 as a solid 8x-compression default
- Quantization adds a compressed copy beside originals; changing datatype (float16/turbo16) rewrites the originals — the difference decides whether anything full-precision survives to rescore against
- Without rescoring, 1-bit recall is only 0.605; oversampling 4 brings it back to 0.988, near the float32 baseline, at 1/32 the vector size
The article lays out a four-step setup: estimate dense vector footprint, pick method and bit depth starting at bits4, pin the quantized copy and keep originals cold, then set rescore/oversampling from measurements on your actual deployment. Hybrid search shares one page cache between dense and sparse reads, so rerun the full query before sizing a deployment.
More from Infra
- AI agent beats NVIDIA's FlashMLA by 9.8% on B200 after 22-hour black-box optimization run — Equivalent-Grass-527 · 2026-11-02
- 20-turn agent loops make $20/mo SaaS seats unprofitable by turn 14 — caching + routing fixes it — TokenMarginDev · 2026-11-02
- "My AI Runs Local": the sticker every AI laptop needs — DavidLinthicum · 2026-11-02
- US-China truce paperwork lags reality: AI supply chain deadlines hit from Nov 9 — demian_ai · 2026-11-02
- swift-transformers 1.4.0 ships with breaking decoding changes from 15 contributors — pcuenq · 2026-11-02
- France leads CVPR 2026 submissions in compute hours, credited to Jean Zay supercomputer — IgorCarron · 2026-11-02