Search 1.4M Images in 15ms: int8 + Rescoring Hits 99.94% Recall with Sentence Transformers

tomaarsen · x · 2026-11-10

A Sentence Transformers maintainer demonstrates an image search setup that queries 1.4M images in 15ms server time. Key details:

Coarse retrieval with int8 quantization followed by cheap fp16/fp32 rescoring largely negates the quantization hit: vs exact fp32 search, int8 alone gets 97.24% recall@100, while int8 + rescoring reaches 99.94%
After rescoring, only the actual images need loading for the user
The approach transfers to any domain; you can easily finetune and deploy with Sentence Transformers, and the author suggests having your agent try it on your own data

More from coding & agent