int8 quantization plus fp16 rescoring: 1.4M embeddings retrieved in under 2ms
tomaarsen · x · 2026-11-10
Sentence Transformers maintainer Tom Aarsen details a two-stage retrieval trick: int8 quantization isn't lossless for recall, but an int8 index first surfaces candidates, then fp16 embeddings loaded from disk are rescored against the fp32 query to produce a new top 30.
It ties into his Hugging Face article on binary and scalar embedding quantization, which covers theory and practice of quantizing embeddings for faster, cheaper retrieval, including a demo over 41 million Wikipedia texts.
More from Infra
- Godzilla and Sonic on silicon: the chip industry's lost die-art easter eggs — thehiphopswami · 2026-11-10
- MLCommons lays out SC26 MLPerf agenda with Training v6.1 results and roadmap session — TheKanter · 2026-11-10
- Cloudflare studies quantum-proof Privacy Pass as forged tokens could bypass bot checks — shashib · 2026-11-10
- Anthropic joins AI's natural gas frenzy with $3.4B turbine purchase — TechCrunch AI · 2026-11-10
- Marvell to buy Lumentum in $115B all-stock deal betting big on optical AI data-center links — thione · 2026-11-10
- Anthropic discloses $16.96B Q3 revenue and $895.4B in compute commitments ahead of IPO — thione · 2026-11-10