Generative retrieval with a 4B model memorizes 200M pages, cutting URL lookup latency to 10ms
Chris__Codes · reddit · 2026-11-19
A Reddit discussion explores generative retrieval as a replacement for vector search for fast URL lookups in search-heavy agents.
- Traditional pipelines (query parsing, candidate retrieval, reranking) add 100ms+ per call; frontier LLMs writing URLs from memory mostly hallucinate dead links
- The approach: a 4B model trained on click data memorizes 200M pages and outputs two-part numeric IDs mapped to a real URL table, bringing latency to 10ms in 10GB of VRAM vs 400GB+ for exact vector search
- Limitation: memorization works well for popular pages, but long-tail and brand-new pages still need a live index fallback
More from coding & agent
- Thorsten Ball: Teams still treat agents as code writers, not robot engineers — RealGeneKim · 2026-11-19
- Agentic engineering's new pattern: one coordinator agent managing all your sessions — devenbhooshan · 2026-11-19
- Redditor plans €300 first test of programmable AI video editing workflow — yeah280 · 2026-11-19
- Critical Vite RCE vulnerability in dev server prompts emergency patched releases — film_girl · 2026-11-19
- 226K custom agents deployed to Slack as Cognition and others build on it — silasalberti · 2026-11-19
- Most Teams Deploy AI Like Hiring Someone and Never Explaining the Company — diegobuildsai · 2026-11-19