Ant Group's Repo-Premium benchmark: BM25 with LLM-written identifiers matches dense code search

hq-bench · hf · 2026-11-20

Ant Group builds Repo-Premium, a realistic repo-level code-search benchmark ranking files in real repositories across six languages, PRs from 2022-2026, and a preregistered pool over 91 repos. Findings: the dense advantage over BM25 largely traces to evaluation choices and queries lacking rare identifiers; expanding queries with LLM-written identifiers (generate-then-grep) lets BM25 match dense retrieval in every Python pool, even on PRs opened after the LLM's release. Guidelines: fair lexical baselines, dense+BM25 fusion, and adding code words to queries.

Original post →

More from coding & agent

coding & agent channel →