Agent Retrieval Bench shows coding agents often fail before patch generation even starts

Bowen Qin · hf · 2026-07-30

A new benchmark, Agent Retrieval Bench, evaluates the upstream step that coding agents often miss: finding the right repository context before patch generation.

What it measures

Four positive-retrieval tasks: code2test, comment2context, trace2code, and edit2ripple
A selective-retrieval subset with no-gold cases and wrong-repository controls
Real coding-workflow signals built against frozen base-commit repositories

Scale and findings

427 samples across 25 repositories
345 positive examples, 50 natural no-gold examples, 32 counterfactual controls
308 base-commit snapshots, 392,000 files, and 7.9 million chunks

Main results

No single retrieval approach dominates across tasks
Qwen3-Embedding-4B gets the best sample-weighted MRR on positive samples
Qwen3-Embedding-8B gets the best Recall@20
RepoMap gives the best budgeted context yield at 8K tokens
Logged agent trajectories miss every gold file on 27%–35% of samples
Selective thresholds calibrated with counterfactuals do not improve success on natural no-gold cases, showing a calibration gap
Retrieval-derived seed context improves file F1 versus random non-gold context, but there is still substantial headroom even with oracle gold context

Original post →

Agent Retrieval Bench shows coding agents often fail before patch generation even starts

More from coding & agent