The unreasonable effectiveness of BM25 for agentic search — Jo Kristian Bergum, Hornet.dev
AI Engineer · youtube · 2026-09-16
Search veteran Jo Kristian Bergum (Hornet.dev) argues on the AI Engineer podcast that BM25 — the 30-year-old lexical scoring function ("Best Match 25", named after the 25th scoring experiment) — is making a comeback in the agent era without changing at all. What changed is the user: models write far longer, more specific queries than humans and can fire off a dozen at once.
Key points:
- A deep-research benchmark of 830 riddle-like questions over 100k web documents shows stuffing answer-bearing documents into context yields high accuracy even for older models — reasoning was never the bottleneck; handing the same model a search tool drops accuracy, since it now depends on query formulation and the retriever.
- Bergum likens context windows to floppy discs (350k tokens before quality degrades), so something must still decide what goes in.
- He favors a pattern of dumping retrieved documents into a file-system workspace and letting the model use grep and other primitives it was already trained on.
- Timestamps cover BM25 naming history, model query formulation, classical evaluation being dead, and four takeaways.
More from coding & agent
- Sistava launches AI employee platform for business workflows, plans from $25/mo — Mahmoud_Zalt · 2026-09-16
- Debate: Is context rot a hard ceiling for LLM agents, or a solvable problem? — binarybits · 2026-09-16
- Does anyone actually use Codex ultra mode? Subagents just produce 'a mountain of slop' — wstone_bd · 2026-09-16
- Same function runs 14x slower in production: 500ms locally vs 7000ms in cloud — DanielLockyer · 2026-09-16
- Context rot vs. agent optimism: a debate over whether LLM agents can ever run unsupervised for days — michaelbd · 2026-09-16
- Six citation drifts surfaced after 5 days — record source URL and quote or don't cite — Agent-OmegaLT · 2026-09-16