Ant Group: LLM-written identifiers let plain BM25 match code embeddings in repo-level search

Embeddings vs. Grep: Rethinking the Dense Advantage in Code Search

Siqiao Xue, Ziyin Zhang, Zihan Liao, Jin Qin, Yixiang Mu, Hang Yu

cs.SE

2026-11-16

Ant Group traces code search's dense advantage to evaluation choices and missing identifiers; with LLM-written anchors, plain BM25 matches or beats embeddings.

What problem this solves

On code-search benchmarks, specialized embedding models beat BM25 by nDCG@10 margins that often exceed 0.1, and teams build embedding indexes on the strength of those numbers. Meanwhile a practitioner report (Augment Code) reached the top open-source spot on SWE-bench Verified with an agent that searched using grep and find, and concluded embeddings were not the bottleneck. Both sides can be right, but no existing benchmark could decompose the dense advantage under one consistent protocol: they pair curated or docstring-style queries with old code, run the lexical baseline on the dense retriever's own 1,000-character chunks, and mix label rules (SWE-bench gold patches contain no test files; PR file lists do). Weak baselines are a known source of false gains in neural ranking; their effect on repository-level code search had never been measured.

Method

Ant Group built Repo-Premium, a file-retrieval benchmark inside single repositories. The input is a natural-language description of a real code change plus the repository at a pinned snapshot; the output is a ranking of files; the gold set is the non-test source files the merged change actually edited, so labels come straight from the PR with no annotator judgment. The core holds 1,265 queries over ten SWE-bench Verified Python repositories (10,200 files); an extension adds 1,560 queries across JavaScript, TypeScript, Go, Rust and Java, plus a preregistered lockbox of 91 repositories. The four core pools share corpus, snapshot and label rule, varying only the query wording (curated statements vs. raw PR text) and the change's date (2022, 2023, 2026, sitting clearly before or after the encoders' training cutoffs).

Three controlled interventions do the diagnosis: switch BM25 from 1,000-character chunks to whole files and strip PR template boilerplate; unify the label rule so edited test files leave the gold set, with both rules reported; and generate-then-grep, where qwen35-122B reads only the stripped query and drafts identifiers, API names, error terms and a code skeleton to prepend for BM25. The generator never sees gold paths or patches. Thirteen dense encoders were evaluated, plus SPLADE and ColBERTv2.

Results

Setting (dense − BM25, nDCG@10)Cur-Pre (curated)Raw-Post (2026 PRs)
Conventional: BM25 on 1,000-char chunks+0.121+0.152
BM25 on whole files+0.034+0.059
Whole files, edited tests counted as relevant+0.165+0.136
Query + LLM-written identifiers−0.081−0.054

Chunking alone decides the sign: on django, dense retrieval loses to whole-file BM25 by −0.153 and flips to +0.119 once both use 1,000-character chunks, because encoders truncate long files while BM25 benefits from large windows. Against whole-file BM25, none of the thirteen encoders keeps a significant premium on the curated or 2026 pools, and seven fall significantly below zero on the curated pool. The test-file bias is behavioral: dense retrieval ranks a test file first on 59% of curated queries, against 34% for tuned BM25. What survives is mostly coverage (Recall@10 up 0.10 to 0.19), concentrated on queries whose rarest token shared with the code is still common: by shared-token MaxIDF quartile, the premium decays from +0.217 to +0.050.

Generated identifiers help BM25 far more than the encoder: +0.11 to +0.19 against +0.03 to +0.07. After expansion, BM25 trails in no Python pool by point estimate and leads the curated pool by a significant 0.081; on PRs opened after the generator's release it still leads by 0.094. The direction holds with Qwen3-Embedding-8B and with five generators from 4B to 122B.

Fixed fusion beat every single choice. Reciprocal rank fusion of dense and whole-file BM25 adds +0.060 over dense alone on development data, and the two fixed fusions added +0.019 and +0.045 on 539 preregistered lockbox queries. BM25 strictly wins 24% of curated queries, yet classifiers, LLM judges and a preregistered router all failed to find them (AUROC 0.47 to 0.54), and the router beat always-fusing by only 0.005. In a 96-task agent evaluation, swapping the search tool (shell with grep, lexical index, dense index, router) moved localization exact-match by at most about 0.04, while swapping the LLM moved it between 0.03 and 0.30.

Why it matters

For anyone building retrieval into coding tools, three rules land directly. Run a fair lexical baseline first (whole-file BM25, stripped boilerplate, one stated label rule) before paying for an embedding index. Keep dense retrieval and fuse it with BM25; do not route per query. When the query names no code, one LLM call drafting the missing identifiers is worth as much as the index. The cost math: one Qwen3-8B call takes about 1.5 seconds, a dense index for a django-sized repository about four minutes to build, so generation is cheaper whenever a repository is searched fewer than roughly 158 times. For agents, spend on the model before the search endpoint. For benchmark builders, this is a complete case study of label rules and baseline settings deciding the sign of a published comparison, and the preregistered-lockbox protocol is worth copying.

Limitations

The authors list: ten Python repositories in the core and one repository per language in the cross-language pools, so modest effects cannot be excluded; BM25's settings were picked on the evaluation queries, making the dense numbers lower bounds; results depend on the generator, with smaller ones closing only two-thirds to 80% of the premium; labels are the files one change edited; function- and line-level targets are untested. Reading closely, a few more caveats stand out: the generator may know these repositories, which the paper admits replaces the premium without explaining it; the MaxIDF stratification is exploratory and was chosen on the same slices, with a per-query correlation of only −0.124; a removed premium means confidence intervals cover zero, and the paper states it did not show equivalence; the agent evaluation measures localization only, with no tests run.

Terms

Source

What people are saying

Related papers

All paper explainers