RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests
Gyuhyeong Kim, Hyojung Gwon, Jeonghyeon Kim, Kyuhong Shim, Sunjae Lee
cs.AI, cs.LG, cs.SE
2026-08-28
RealSWE shows realistic sparse requests drop resolution 6.4 points on average; stating desired behavior recovers about 8 points on bugs, reproduction steps add almost none.
SWE-bench scores coding agents on curated GitHub issues: long, templated, and packed with reproduction steps, environment notes, and a statement of desired behavior. Real users talk in short, casual lines. "Crashes on empty input, fix it" can name the same bug as a fully filled issue while leaving the agent far less to go on.
Sungkyunkwan University labels 718 first-turn user prompts from SWE-chat and the problem statements in SWE-bench Verified and Pro along a six-field information taxonomy and four linguistic dimensions. Requests that carry only a problem statement, or that plus leftover extras such as URLs and dates, are 88% of real prompts and 7% of the benchmark. 87% of real prompts are casual; 94% of benchmark problems are formal. Desired behavior appears in 5.4% of real bug-fix prompts and 73.5% of benchmark ones. Prior "realistic" evals are either closed (CursorBench) or mutate issues by truncation and noise, so they cannot isolate which missing field hurts and which tone change does not.
RealSWE turns each executable task into a family of variants that share the gold patch and vary only information mix and linguistic style. A three-stage GPT-5.4 pipeline classifies bug fix versus feature request, splits the original text into taxonomy fields without rewriting technical content, then rephrases into the majority SWE-chat style while keeping code, tracebacks, and paths intact.
Of 1,229 source problems, 403 contain every required field (five for bugs, three for features). Two annotators each label 100 items; task-type accuracy is 0.95. Decomposition and rephrasing are audited by GPT-5.6 Terra as judge, with human agreement of 0.97 and 0.99 accuracy. 22 candidates that fail a critical rubric are dropped, leaving 381 families (192 bug fixes, 189 feature requests). The kept set is not the easy slice: DeepSeek V4 Pro resolves 53.9% of selected originals versus 63.0% of the excluded 848, and selected patches are larger (median 62 lines versus 38).
Two releases. RealSWE-bench samples one variant per family to match SWE-chat: 74% problem-only and 26% problem-plus-extras for bugs, 72%/28% for features, 381 tasks. Mean description length is 1,417 characters against SWE-chat's 1,427, well below the 1,672–2,776 range of Verified, Pro, and DeepSWE. RealSWE-framework exposes every variant for custom mixes.
Seven models run on mini-SWE-agent v2, Bash-only, 100-step cap, three seeds: DeepSeek V4 Pro and Flash, MiMo V2.5 Pro and V2.5, Claude Haiku 4.5, Qwen3.7 Plus, MiniMax M3.
All seven lose resolution on RealSWE-bench, 6.4 points on average, 10.3% to 16.2% relative to each model's original score (mean 13.6%).
| Model | Original | RealSWE | Delta |
| DeepSeek V4 Pro | 53.9% | 45.9% | −8.0 |
| DeepSeek V4 Flash | 49.7% | 41.6% | −8.0 |
| Qwen3.7 Plus | 50.1% | 43.5% | −6.6 |
| MiMo V2.5 Pro | 49.1% | 44.0% | −5.1 |
| MiMo V2.5 | 48.4% | 40.7% | −7.7 |
| Claude Haiku 4.5 | 42.1% | 36.7% | −5.4 |
| MiniMax M3 | 34.1% | 30.1% | −4.0 |
Rankings move. MiMo V2.5 Pro climbs from fourth on the originals to second, overtaking Qwen3.7 Plus and DeepSeek V4 Flash. The MiMo–Qwen gap shift is +3.7 points (95% CI [+0.7, +7.3]), and MiMo is about 2.5× cheaper per task (6.5 versus 16.1 cents). Bug-fix resolution falls 9.1 points on average; feature requests fall 3.7. Six of seven models spend a bit more (cost +6.2%, steps +1.8% on average). Extra exploration does not buy back the lost solves.
Field ablations locate the cause. Linguistic style barely matters: all-fields rephrasing changes bug-fix rates by 0.0 points on average and feature requests by −1.8, with none of eight contrasts significant. Stripping extras, environment, and reproduction steps moves the rate by 1.8 points on average. Removing desired behavior [D] drops all four models 7.1 to 8.9 points (8.0 average, Holm-adjusted p<0.01). Removing motivation [M] on feature requests drops 3.4 points on average, 7.1 for DeepSeek V4 Flash. The [P]/[PA] mix that real users actually send is exactly the mix that omits those two high-value fields.
Leaderboards built on polished issues overstate performance on short user requests, and they can steer buyers toward a more expensive model that is not better under realistic inputs. RealSWE is the same task under a different brief, not a harder repository suite.
The product advice is narrow. For a bug, say how it should behave after the fix. For a feature, say why it is needed. Reproduction steps and environment notes add tokens along this ablation path and almost no solves, yet they are the fields benchmarks supply and users skip. Agents can ask those two questions before editing, or infer them from the repo and write them into the spec.
The model set omits current SWE leaders such as GPT-5.6, Opus 5, and Kimi K3, so the gaps may shrink at the frontier. Evaluation is single-turn, as in SWE-bench: the agent cannot clarify, and sparse requests may hurt less in a real conversation. Requiring complete fields cuts 1,229 tasks to 381. The authors show the kept set is not easier and not a single-repo cluster, but it is still a subset. Decomposition and rephrasing are LLM-driven with sampled human checks; judge–human macro-F1 on rephrasing is only 0.75. The 718 SWE-chat prompts are first turns from people already talking to a coding agent, not every engineer's issue-writing habit.