int8 embeddings plus Usearch index: 1.4M vectors searched in under 2ms on CPU
tomaarsen · x · 2026-11-10
Tom Aarsen shows the quantization step of his low-latency retrieval pipeline: after quantizing embeddings from fp32 to int8, an efficient Usearch approximate nearest-neighbor index retrieves across 1.4M embeddings in under 2ms on a tiny CPU. A screenshot shows a "bookcase" query served in 13.8ms total (10ms embed + 1.4ms retrieve).
Combined with TinyCLIP finetuning, Matryoshka truncation to 256d, this stack delivers 20ms image retrieval on a tiny CPU.
More from Infra
- UC Berkeley open-sources FreeToken, an edge-native engine running frontier MoE models on consumer hardware — petrusenko_max · 2026-11-10
- Zero data retention is why enterprises self-host: Azure ZDR requires approval, no ZDR for fable — HanchungLee · 2026-11-10
- NVIDIA's CPU-ARC: reasoning caches 16–291x cheaper with frontier-level performance — nvidia · 2026-11-10
- One slide: when to use Agent Sandbox vs Agent Substrate for agent infrastructure — davemccollough · 2026-11-10
- Godzilla and Sonic on silicon: the chip industry's lost die-art easter eggs — thehiphopswami · 2026-11-10
- MLCommons lays out SC26 MLPerf agenda with Training v6.1 results and roadmap session — TheKanter · 2026-11-10