A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents
Xin Zhou, Chun Yong Chong, Kisub Kim, Yun Peng, Rui Shu, Zihan Wu, Xu Han, Guowen Yuan, Zeyang Zhuang, Jounghoon Kim, Jeongjin Ju, Seongmin Ju, Taein Yoon, David Lo
cs.SE, cs.AI
2026-08-10
SWE-RPG tests Claude Code, Codex and OpenCode on 163 Python/Java PR tasks: mean resolve 31.5% (best 49.7%), with requirement failures in 24.5-46.0% of runs.
Most repository-level coding-agent benchmarks still reduce a run to whether the final patch passes tests. Getting from a raw issue to a correct change requires recovering constraints the ticket left implicit, planning edits against the actual tree, then writing code. A pass/fail bit cannot say which of those stages broke.
RACE-bench and Dialogue SWE-Bench start inspecting intermediate work. The first does not supply gold for implementation-critical details missing from the original request. The second hides details on purpose and studies recovery through simulated dialogue. SWE-RPG asks a sharper diagnostic: when the agent fails, did it fail at clarification, planning, or coding.
The set has 163 tasks from 31 mature Python and Java repositories: 113 bug fixes, 50 feature additions (85 Java, 78 Python). Construction is selective: 2,000+ issue-PR pairs down to 400+ runnable images, 200+ with stable tests, 163 after intermediate-GT review. Mean codebase size is 272.5K LOC (max 1.59M). Gold patches average 56.7 LOC; each task ships 2,249 tests on average.
Clarification gold comes from seminar-style interviews with ten engineers at Fortune Global 500 tech firms, then a six-category QA taxonomy: functional intent, business semantics, technical context, interface/protocol, structure and naming, data-structure semantics. Mean 4.53 implicit points per task. Planning gold is an ordered list of actionable steps (mean 2.06 steps, 11.80 constraints). Each step is accepted only if a coding agent following it can produce a patch functionally equivalent to that step's gold subpatch. Two authors review every retained reference. Synthesis used GPT-5.4.
The eval matrix is three scaffolds times six backends, two runs per configuration: Claude Code, Codex, OpenCode, against Claude-Sonnet-5, DeepSeek-V4-Pro, GLM-5.2, GPT-5.6-Terra, MiniMax-M3, and Kimi-K3. Resolve follows SWE-bench: apply, all fail-to-pass tests green, no pass-to-pass regressions. GPT-5.6-Sol attributes each failure to the earliest diverging stage; 46 of 50 stratified samples match human consensus (92%). Coverage labels agree on 96% of a 50-point sample.
Mean resolve rate across 18 configurations is 31.5%. Best pair is OpenCode with Kimi-K3 at 49.7%, then Claude Code with Kimi-K3 at 49.1%. Worst is Codex with MiniMax-M3 at 17.8%. Scaffold means: OpenCode 33.0%, Claude Code 32.8%, Codex 28.5%. Backend means: Kimi-K3 46.6%, DeepSeek-V4-Pro 38.7%, the rest 22.3–29.2%. Average cost is $1.59 per task and 8.5 minutes. OpenCode plus Kimi-K3 sits above the resolve mean at below-average cost; Claude Code with the same backend is more expensive. Longer runs do not reliably produce better patches.
Requirement failures are the largest slice for most configs, 24.5–46.0% of all runs. Code generation is 7.4–37.4%, planning 5.5–17.8%. 46.7% of runs die in clarification or planning. Close totals can hide opposite profiles: Codex and Claude Code with DeepSeek-V4-Pro resolve 40.5% and 39.3%, but requirement failures are 26.4% vs 40.5% and implementation/verification failures 23.9% vs 7.4%.
Planning coverage falls from location to approach to constraints/validation: Claude Code 79.7% to 64.5% to 41.6%, Codex 65.6% to 49.0% to 37.1%, OpenCode 43.8% to 31.1% to 24.8%. Clarification is stronger on intent and scope than on interfaces, conventions, and data semantics (Claude Code structure 54.2%; Codex structure 42.0%, data semantics 41.9%).
This is a diagnostic set for people who build coding agents. When two configs land near the same resolve rate, the fix may be implicit-constraint recovery in one case and verification obligations in the plan in the other. The size of the requirement slice matches a familiar engineering fact: tickets rarely spell out "touch TSV only, leave CSV alone."
The set is much smaller than SWE-bench Verified and only covers Python and Java. It is usable as a training or regression signal. It is too thin to declare a scaffold universally stronger.
The authors list the obvious ones: 163 tasks, 31 repos, Python and Java only; no human-in-the-loop; GTs and stage labels are LLM-assisted even though humans review them; no inter-annotator agreement is reported. The judge was checked on 50 samples. Gold is reverse-engineered from merged PRs, so it carries hindsight the original developers may not have had up front. Plans average 2.06 steps, which may under-represent large changes. The model matrix is 2026 commercial systems; smaller open models are absent.