SWE-Touch: Benchmarking Coding Agents When Users Touch the Code
Yuqiao Tan, Jinxiang Meng, Fangyu Lei, Minzheng Wang, Shizhu He, Jun Zhao, Kang Liu
cs.SE, cs.AI, cs.CL
2026-08-04
SWE-Touch injects task-conflicting user code edits into ongoing agent repairs; across 9 models on SWE-bench Verified, average resolve rate drops 7.7 points, with losses from 1.3 to 16.5.
Repository-level benchmarks like SWE-bench score agents that work on a codebase alone, and the interactive variants that followed (Ambig-SWE, HiL-Bench, SWE-Interact, SWE-Together) let users participate only through messages. Real development is messier: the user and the agent share one repository, one set of files, one executable state. The authors' analysis of the released SWE-chat logs finds that 59.0% of real sessions include repository changes made by the user. So "the user edits the code" is a frequent interaction channel that no coding benchmark has actually stress-tested. SWE-Touch asks the direct question: when a user modifies the shared codebase mid-task, can a coding agent notice, reconcile the conflict, and re-validate the affected behavior?
Three pieces. First, mining task-critical regions. For each task the authors run three models from different families (GPT 5.5, GLM 5.1, MiniMax M2.7) and intersect, across trajectories, the regions each one read and the regions each one edited, kept separate. Regions are then selected by priority (edits over reads, implementation files over tests), capped at eight per task. Of 200 tasks, 174 yield edit-based regions, 24 read-based, and 2 fall back to the reference patch.
Second, building the Counter-Edit. A separate GPT 5.5-backed User Patch Generator inspects code around the critical regions and writes a small, locally plausible unified diff that touches only implementation code. Every candidate is replayed against the fail-to-pass tests and must satisfy three conditions: the edit alone does not solve the task, the reference patch alone does, and the two combined still fails. Of 192 code edits, 150 pass full three-state validation; in the remaining 42 the reference patch no longer applies cleanly after the edit but the conflict holds, and they are kept. Counter-Edits on SWE-bench Verified average 7.0 changed lines across 1.04 files, engineered to read like a confident developer's plausible mistake.
Third, delivery and control. The runtime watches the agent's actions and injects the patch whenever it touches code overlapping the patch region, up to K=3 times by default, each time paired with a contextual message from GPT-4o. The user persona is a "Miscalibrated Builder": sincerely convinced the edit is right, escalating from collaborative to insistent across stages. The Co-Edit control uses the same machinery to produce a task-aligned helpful patch that alone still does not solve the task. The whole design forces the agent to detect external change, reconcile the conflict with the task, and re-run targeted tests.
Across nine models on SWE-bench Verified (200 tasks, three runs each), Counter-Edit cuts average resolve rate by 7.7 points; every model is negative, with losses from 1.3 to 16.5.
| Model | Vanilla | Counter-Edit | Δ | Retention |
| Claude Opus 4.8 | 85.2 | 83.3 | -1.8 | 96.0% |
| GPT 5.5 | 80.5 | 79.2 | -1.3 | 95.0% |
| Qwen 3.7 Max | 75.2 | 70.3 | -4.8 | 90.3% |
| GLM 5.1 | 72.7 | 68.3 | -4.3 | 83.3% |
| MiniMax M2.5 | 75.7 | 66.2 | -9.5 | 78.3% |
| Kimi K2.6 | 70.3 | 64.3 | -6.0 | 87.2% |
| DeepSeek V4 Pro | 74.8 | 63.8 | -11.0 | 81.5% |
| MiniMax M2.7 | 76.5 | 62.7 | -13.8 | 78.1% |
| Qwen3-Coder-480B | 57.2 | 40.7 | -16.5 | 60.8% |
The two frontier agents stay on top; the open-source mid-tier reshuffles badly. MiniMax M2.7 falls from rank 3 to 8, while Qwen 3.7 Max climbs to rank 3. Degradation persists on the longer-horizon SWE-Bench Pro and DeepSWE extensions (about -4.9 and -3.4 on average). Ablations: message-only is small and inconsistent (-2.0 to +3.0); a silent code edit alone hurts every model (up to -9.5); on seven models Co-Edit averages only -0.1, proving the difficulty is conflict, not interruption. Failure analysis of 526 solved-to-unresolved runs finds 63.3% terminate with the user's conflicting code still active, and the rate at which a model revises or removes the edit correlates with its resolve-rate loss at Spearman ρ=0.80: Claude Opus 4.8 revises in 79.3% of failures, DeepSeek V4 Pro in only 15.9%.
For anyone pairing with Cursor, Trae, or Copilot Workspace, this is the daily reality: you watch, you tweak a couple of lines, the agent has to keep up. The paper puts numbers on a gap that practitioners already feel: a strong autonomous leaderboard score does not guarantee robustness in a shared workspace, and several open-source models that look competitive in isolation degrade substantially once a user touches the code. When choosing an agent for real collaborative development, retention under user edits deserves to sit next to plain pass rate.
The authors flag three. The benchmark covers one controlled class of task-conflicting edits, not the full distribution of collaborative behavior (partially correct fixes, requirement changes, complementary edits). Region-based triggering is chosen for comparability, where real users intervene dynamically based on what the agent is doing. And the environment is meant to seed training of collaboration-aware agents rather than only evaluate them. My own concerns: the user persona is fixed as "always wrong and confident," which is a narrow slice of real users; the 200-task subset includes 8 text-only fallbacks and 6.4% of runs where the patch never landed, both kept in the scored set; the longer-horizon extensions use only 25 tasks and two runs each, so variance is high; and all evaluated models are mid-2026 versions, so the ranking will age as models improve.