Asymmetric retrieval: index with 9B, encode queries with 0.6B for frontier accuracy at low latency
antoine_chaffin · x · 2026-10-08
The author explains why the setup matters: search is asymmetric — indexing is offline and amortized over the index lifetime, while query encoding is online and latency-critical. So you can index your corpus with the 9B model and encode live queries with the 0.6B, or encode private documents on-device with the 0.6B and merge them into a 9B cloud index.
Related event: Perplexity Open-Sources Multimodal Embedding Models pplx-embed-v2-late(38 posts)→
More from Infra
- Tracking a 24/7 agent for 30 days: $6 VPS, $22 API, and uptime is the real leak — YamOk7317 · 2026-10-08
- Bain: nearly 150GW of new data center capacity by 2030, requiring $5-6.5 trillion — Beth_Kindig · 2026-10-08
- TypeSafeAI's Jev served a trillion tokens on Modal within three days of launch — josh_wills · 2026-10-08
- llama.cpp PR adds GPU cache for host-resident MoE experts, big speedup potential — jacek2023 · 2026-10-08
- Mapping the firm-power stack behind Google's massive nuclear deal: CEG, TLN, VST, Hubbell, Bloom — demian_ai · 2026-10-08
- Why temp=0 LLM inference still isn't deterministic: floating-point order and parallelism — ducha_aiki · 2026-10-08