Ant Group's Repo-Premium benchmark: BM25 with LLM-written identifiers matches dense code search
hq-bench · hf · 2026-11-20
Ant Group builds Repo-Premium, a realistic repo-level code-search benchmark ranking files in real repositories across six languages, PRs from 2022-2026, and a preregistered pool over 91 repos. Findings: the dense advantage over BM25 largely traces to evaluation choices and queries lacking rare identifiers; expanding queries with LLM-written identifiers (generate-then-grep) lets BM25 match dense retrieval in every Python pool, even on PRs opened after the LLM's release. Guidelines: fair lexical baselines, dense+BM25 fusion, and adding code words to queries.
More from coding & agent
- Omnigent 0.23.0 ships session grids, /rewind for Claude chats, and cheaper automation polling — matei_zaharia · 2026-11-20
- Developer gives Claude control of his sheep enclosure and it now herds sheep autonomously — repligate · 2026-11-20
- Neon's MCP server now executes SQL as a selected Postgres role, constraining agents via DB grants — JeremyCMorgan · 2026-11-20
- Even with OpenAI's Daybreak access, Codex still blocks cybersecurity requests — AIFlow_ML · 2026-11-20
- Codex Community Meetup Singapore set for Nov 20 with 392 already registered — gabrielchua · 2026-11-20
- Dev reverse-engineers Siri's UI, swaps in a local LLM — and hijacks SpringBoard to steal the real Siri Orb — Gluxin · 2026-11-20