Ant Group's Repo-Premium shows BM25 matches dense retrieval in fair code-search tests
_reachsumit · x · 2026-11-18
Ant Group released Repo-Premium, a realistic repository-level code-search benchmark: a preregistered test pool across 91 real repos with curated and raw queries in Python, JavaScript, TypeScript, Go, Rust and Java, and pull requests from 2022 to 2026.
The study finds dense retrieval's lead over BM25 on existing benchmarks is inflated by two factors:
- Evaluation choices that favor dense retrieval, such as splitting files into chunks for BM25 and counting edited tests as relevant;
- Queries lacking the code's rare identifiers.
Expanding queries with LLM-written identifiers (generate-then-grep) helps BM25 far more than dense retrieval, letting it match dense retrieval in every Python pool—even on PRs opened after the LLM's release. No method the authors tried predicts which retriever wins on a given query. They derive three guidelines: benchmark dense gains against a fair lexical baseline, fuse dense retrieval with BM25, and add the code's words to queries that lack them.
More from coding & agent
- Scoble Tests Hy4 Agent in WorkBuddy on a 20-Step Real Office Job — Scobleizer · 2026-11-18
- DetouR follows hyperlinks for multi-hop QA, adding 4.0 EM on 2Wiki with zero extra calls — _reachsumit · 2026-11-18
- Echoes fool RL search agents: 8 paraphrased false claims overturn 58-73% of correct answers — _reachsumit · 2026-11-18
- rust-anything rewrites legacy projects to Rust with AI, up to 20x faster — aigclink · 2026-11-18
- Jetriever ranks agent memories by utility not similarity, hitting gold in 84.8% of cases — _reachsumit · 2026-11-18
- Building a Claude skill for anime-style emotional framing techniques — andrew_n_carr · 2026-11-18