Quantizing your RAG index? Recall fell from 0.992 to 0.055 on the same method
Plane-Preference6245 · reddit · 2026-11-02
When moving a RAG index to binary/halfvec to save memory, the author found compression loss depends heavily on your embeddings: binary + rerank kept 0.992 of true top-10 recall on OpenAI embeddings, but only 0.348 on GloVe and 0.055 on SIFT — same method, same settings.
He open-sourced shrinkvec (MIT):
- Feed it your vectors (.npy/.parquet or a Postgres table); it holds out queries and computes exact neighbors
- Benchmarks 13 options (halfvec, int8, binary, RaBitQ, TurboQuant, rerank pipelines) for recall, size and latency, then picks the smallest one hitting your target
- Best practice: run it in CI with --min-recall 0.95 so the build fails if switching embedding models quietly hurts search
Pip install and Docker demo available.
More from Infra
- FT chart: Big Tech's surging CapEx is a rational bet, not reckless spending — luisdans · 2026-11-02
- Liquid AI's 2B d1 decision models beat same-size general models on classification benchmarks — JosephJacks_ · 2026-11-02
- Intelligence got 100x cheaper, so usage exploded 500x: Jevons Paradox in action — luisdans · 2026-11-02
- OpenAI may rebrand its data centers as 'AI compute campuses' — TechCrunch AI · 2026-11-02
- $300B Louisiana project pipeline squeezes steel supply, H-beam lead times hit 9 months — MatthewChang · 2026-11-02
- Benchmark: Speedy inference engine crushes Avalon — 1.6x faster reads on Qwen3 long context — MotorNetwork380 · 2026-11-02