Company Knowledge Bench: real-world retrieval benchmark tests 7 retrievers
CShorten30 · x · 2026-10-03
kapa.ai's team built Company Knowledge Bench, the first retrieval benchmark on messy, real-world company knowledge, from 1,000 real queries by humans and agents. Early results across 7 retrievers: grep works surprisingly well but is 5x slower and floods context with 40K tokens per query; optimized agentic retrievers do far better — Kapa Deep scored highest (0.65) in 5s with an eighth of the tokens.
More from coding & agent
- LangSmith launches Custom Apps: prompt-built custom UIs on your agent data — LangChain · 2026-10-03
- Decade-long dev cuts a full video with AI agents, no editor, in one week — john__allard · 2026-10-03
- Tip: Use Apple Create ML to Train a Local Text Classifier for Product Categorization — airesearch12 · 2026-10-03
- GPT-6-powered agent clears all WoW starting-zone quests in 40 minutes, 0 deaths — petrusenko_max · 2026-10-03
- Turning an iPhone into a second GPU for a MacBook: 44% faster prefill on Qwen 27B — StayLameBro · 2026-10-03
- One image prompt gets Claude to build a playable browser game plus its trailer — justin_hart · 2026-10-03