Boosting Coding Agents: BM25F Pre-Retrieval Hits 77.9% Top-20 Accuracy
newtophillyfromkc · reddit · 2026-08-06
The author built a local fast-retrieval tool called prepass to solve the time-consuming initial search phase that coding agents like Claude Code or Codex face in unfamiliar repositories.
Methodology & Testing
- Benchmarked against 240 real GitHub issues from SWE-bench Lite.
- Uses the classic BM25F lexical ranking algorithm (covering file contents, filenames, and directories) via a single ripgrep pass. No embeddings or API keys required.
- The tool injects only matched file paths into the agent, never the file contents.
Key Results & Findings
- Performance: Achieved 77.9% accuracy in hitting the right file within the top 20, 35.4% first-hit rate, and an MRR of 0.475. Median time is only 105ms.
- Documentation Noise: Surprisingly found that in-repo documentation (like Django's embedded docs) dominated lexical matches, taking up 60% of the shortlist. Discounting documentation boosted MRR from 0.383 to 0.475.
- What Didn't Work: Local embeddings, pseudo-relevance feedback, and capping query terms all failed to provide positive gains.
The tool is open-source (Apache-2.0) and works with Claude Code and Codex.
More from coding & agent
- Liquid AI Launches LFM2.5-2.6B: An On-Device Agentic Model — JosephJacks_ · 2026-08-06
- Running Local Agents with LFM2.5-2.6B: A Step-by-Step Guide — JosephJacks_ · 2026-08-06
- Resurrecting a 2015 AI Assistant with ESP32 and a Rotary Display — viksit · 2026-08-06
- Codex One-Shots a Mac App to Triage 1.5TB of Cycling Footage — john__allard · 2026-08-06
- Developer Shares Practical Workflow OS for Claude Code and Codex — thatroblennon · 2026-08-06
- Smart LLM Routing Cuts Inference Costs by 65% While Maintaining 99.2% Accuracy — shensi · 2026-08-06