Practical Code RAG at Scale: Task-Aware Retrieval Design Choices under Compute Budgets
Timur Galimzyanov, Olga Kolomyttseva, Egor Bogomolov
cs.LG, cs.AI, cs.IR
2025-10-23
On Long Code Arena, BM25 with word splitting is the practical winner for code completion, while Voyage-3-Code reaches 0.72 NDCG on bug localization versus 0.57 for BM25.
Code RAG has too many knobs: whole files versus line windows, BM25 versus dense encoders, word versus BPE splitting, syntax-aware chunking or not. Method papers usually sell one setting. What is missing is a bake-off under a shared pipeline and a real compute budget.
JetBrains Research fills that table with two Long Code Arena tasks. Code completion is PL→PL: predict the next line, which refers to a class or method defined in another file. Bug localization is NL→PL: rank files given an issue report.
Retrieval is factored into three modules. A chunker emits whole files, fixed line windows of 8/16/32/64/128 lines, or LangChain syntax-aware recursive splits. A splitter (sparse only) turns a chunk into a bag of lines, words, or BPE tokens. A scorer covers IoU, BM25, the E5 family, the Voyage-3 family, DraCo dataflow graphs, and a directory-distance heuristic.
Completion is generated with DeepSeek-Coder-1.3B; the metric is next-line exact match. Packing budgets are 128, 4096, 8192, and 16384 tokens. Bug localization ranks files with NDCG on Java, Kotlin, and Python. A four-stage search avoids a full grid: pick scorer and splitter on whole files, sweep chunk size, compare syntax-aware splits, then hybridize structure-aware lists with the best sparse or dense retriever.
On completion, BM25 with word splitting is the default quality-latency pick. Exact match is 0.55/0.60 at 4K/16K context, versus 0.39/0.52 for E5-large, about 10 points higher. Word and BPE splitting tie on quality; word indexing is about 9× faster. Line windows slightly beat syntax-aware splits. 32–64 line chunks win at ≤4K tokens; whole files catch up at 16K. DraCo lags chunked retrieval at medium windows and only ties at large ones. Path distance stays worse.
Bug localization flips the ranking. Voyage-3-Code (512-token input) reaches mean NDCG 0.717, E5-large 0.590, BM25-word 0.574. Python narrows the gap: BM25 0.635 versus E5-large 0.606. Latency is an order of magnitude apart: BM25-word 0.07 s per million symbols, E5-large 2.8 s, Voyage-3-Code about 19 s.
Completion latency spans about 180×: IoU-line 0.02 s per million symbols versus 3.3 s for E5-large. On a 2.3 million symbol repo that is roughly 1.2 ms for path distance, 0.5 s for BM25-word, and 7.5 s for E5-large.
| Setting | Metric | Best / baseline |
| Completion 4K/16K | exact match | BM25-word 0.55/0.60 vs E5-large 0.39/0.52 |
| Bug localization | mean NDCG | Voyage-3-Code 0.717 vs BM25-word 0.574 |
| Latency | s / 1M symbols | IoU-line 0.02 vs E5-large 3.3 |
This is a configuration guide, not a new retriever. Start with BM25 and word splitting for code-to-code. Pay for dense encoders when the query is natural language. Grow chunk size with the context window, and do not reach for AST splitting first. For interactive completion, IoU-line is the cheap fallback.
The result is task-dependent. Completion lives on lexical overlap; bug localization needs cross-modal alignment. Those two jobs should not share one index by default.
No deduplication or context compression, so small-window models cannot pack larger spans. Generation uses only DeepSeek-Coder-1.3B; larger variants are said to follow the same trend without a full table. The tasks stop at single-line completion and file-level bug ranking. Dense latency includes query-time encoding, so a precomputed index would look faster. Voyage numbers come from a proprietary API under the paper's batch limits.