Local CPU vector mesh beats BM25 routing 100k tools: 70.7% top-1 at 9.6ms
Key_Coconut_9269 · reddit · 2026-10-27
Why dump 4,000+ tokens of tool schemas into every LLM turn just for an L7 routing lookup? The author decoupled tool discovery from the LLM using a local CPU vector mesh (all-MiniLM-L6-v2 + chromadb HNSW) plus a split execution daemon. Key trick: embed tool name, description, schema keys, and tags as one profile vector instead of the name alone. Cold-cache benchmark on 100,000 synthetic tool endpoints: naive keyword scan 38.1% top-1 at 46ms; BM25 40.4% at 194ms; local vector mesh 70.7% at 9.56ms (p95 11.4ms); in-memory fast path at 0.082ms.
More from coding & agent
- Vercel engineer shuding reimagines UI as complex-valued functions: viewport 720+42i px — shuding · 2026-10-27
- shuding: integrate your UI over real traffic to get the exact average page users see — shuding · 2026-10-27
- shuding takes the Fourier transform of a responsive dashboard as a frequency signal — shuding · 2026-10-27
- shuding mathematically proves UIs never break across infinite prop combos via interval arithmetic — shuding · 2026-10-27
- Filter-then-locate beats whole-file prompts: testing 4 ways to update READMEs with LLMs — KhuyenTran16 · 2026-10-27
- Qiaomu Clipper launches site: open-source Chrome clipper that saves reading notes to Obsidian — vista8 · 2026-10-27