Ant Group's Repo-Premium shows BM25 matches dense retrieval in fair code-search tests

_reachsumit · x · 2026-11-18

Ant Group released Repo-Premium, a realistic repository-level code-search benchmark: a preregistered test pool across 91 real repos with curated and raw queries in Python, JavaScript, TypeScript, Go, Rust and Java, and pull requests from 2022 to 2026.

The study finds dense retrieval's lead over BM25 on existing benchmarks is inflated by two factors:

Expanding queries with LLM-written identifiers (generate-then-grep) helps BM25 far more than dense retrieval, letting it match dense retrieval in every Python pool—even on PRs opened after the LLM's release. No method the authors tried predicts which retriever wins on a given query. They derive three guidelines: benchmark dense gains against a fair lexical baseline, fuse dense retrieval with BM25, and add the code's words to queries that lack them.

Original post →

More from coding & agent

coding & agent channel →