Reformulate, Retrieve, Localize: Agents for Repository-Level Bug Localization
Genevieve Caumartin, Glaucia Melo
cs.SE, cs.AI, cs.IR
2025-12-08
A non-fine-tuned Qwen extracts identifiers and snippets for BM25. MAP@1 rises 36% on LCA; the SWE-Lite agent reaches 0.727 MAP@1 versus 0.476 for BM25.
File-level bug localization still dies on the lexical gap: reporters write in user language, code lives in identifiers, and raw BM25 swallows template boilerplate and stack-trace noise with the useful tokens. Query reformulation edits the query, not the index. LLMs can already extract signals from a bug report; how that extraction changes an agent loop had not been measured on its own.
This paper treats reformulation as an intermediate representation, then measures the lift on BM25 and on a downstream agent ranker separately.
RQ1 touches only the query. Qwen3-30B at temperature 0 fills a JSON schema: explanation (a de-templated summary), paths, filenames, identifiers, snippets, stack traces, and error messages. Pyserini BM25 indexes source files with defaults. The baseline is the unmodified report. Five ablations: full schema, explanation only, all code signals, identifiers plus snippets, and explanation plus identifiers plus snippets. Paths and filenames stay out of the BM25 groups and are reserved for the agent.
RQ2 is a small agent loop: extract, BM25 top-{10,20,30}, view files (headers and imports stripped, then 512 tokens), emit a ranked file list. Tool calls are JSON in the prompt, not native function calling. Invalid JSON and bad paths get up to three self-corrections. A second, fresh-context pass re-ranks the final list. All-at-top keeps the full extraction at the top of the conversation; Best-at-top keeps only the RQ1-winning fields. Models are non-fine-tuned Qwen2.5-Coder-32B and Qwen3-Coder-30B, 16K context, local Mac Studio, three runs averaged.
Data: Long Code Arena bug localization, 150 tasks (50 each in Python, Java, Kotlin, half multi-file); SWE-bench Lite, 300 single-file Python tasks. Metrics: MAP and Hit@K.
On LCA, raw BM25 MAP@1 is 0.158. Identifiers plus snippets reach 0.247 (+36%), Hit@1 0.320 to 0.447. Explanation plus identifiers plus snippets wins MAP@5/10 and Hit@5/10 (0.339/0.359, 0.733/0.813). Explanation alone does nothing (MAP@1 0.154). Stack traces and error messages add noise at larger k.
SWE-Lite starts higher (MAP@1 0.400). The same best mix reaches 0.476, Hit@10 0.753 to 0.853. The authors attribute the smaller gap to better-aligned issues.
The SWE agent with Best-at-top Qwen3-30B hits 0.727 MAP@1 / Hit@1, 35% above the best BM25 0.476. All-at-top reaches Hit@10 0.928. Published file-level numbers: SWE-agent (GPT-4) 0.573, Agentless (GPT-4o) 0.697, fine-tuned LocAgent 0.759. A non-fine-tuned 30B-class model clears SWE-agent and Agentless and still trails LocAgent. On LCA, Qwen2.5 All-at-top MAP@1 moves from 0.230 to 0.317. Run-to-run standard deviation stays at or below 0.0075.
Extraction takes 4.4–5.1 s; a full localization run is about 18–78 s. Tool traces are messy: 20–40 viewfile calls per repo collapse to 1.6–5.3 unique files after dropping duplicates and bad paths.
If graphs and fine-tuning are off the table, compressing a bug report into summary plus identifiers plus snippets before BM25 is the cheap first hop. On LCA's mixed-language, multi-file set that hop is worth 36 MAP@1 points. Letting the agent open a few files lifts SWE first-file ranking further.
This is not a new SOTA. Fine-tuned LocAgent still leads. The useful split is isolating query rewrite from graphs and fine-tuning, which is what you want for a low-cost IDE helper.
File-level only; method-level chunking is future work. SWE-Lite gold is still one file. Pretraining on web-scale code may include these projects, so numbers are an upper bound. Extraction quality is tied to Qwen3-30B. Best-at-top omits paths and filenames, which may cap ranking at larger k. The 16K window and 512-token preview only show the top of long files. JSON and path errors are patched by retries; Qwen3 also times out.