WideSWE: Can Coding Agents Coordinate Changes Across Repositories?
Baoyi Wang, Xingliang Wang, Jinyang Wu, Keming Wu, Chen Zhi, Jianwei Yin
cs.SE
2026-09-27
120 cross-repo tasks from 41 ecosystems. Seven agents succeed fully on 10.83%-42.50%; Codex CLI + GPT-5.6-sol leads. Split runs recover omissions, rarely fix bad edits.
SWE-bench scores agents on real GitHub issues inside one repository. Later suites stretch the horizon, but the unit of success is still a single codebase. Product work often is not. Adding a shared tracing policy to Sentry's SDKs means coordinated edits in Go, Python, and Ruby. A new PHP SDK capability also has to land in two downstream repos that depend on it.
Across 103 ecosystems, 109,233 of 1,729,171 merged PRs explicitly point at another repo in the same org. The question is whether a coding agent can find every repo that must change, finish the shared request, and keep behavior consistent across those repos.
A WideSWE task is one feature or bug fix that needs substantive edits in at least two target repositories. The agent gets a merged prompt and a historical workspace. Target repos are scored. Up to 20 extra context repos can be read or edited and do not count toward success. Gold patches and hidden tests stay hidden.
From the 200 GitHub orgs with the most total stars (Gitstar Ranking, 5 June 2026), the authors keep 103 active ecosystems. They mine PRs merged after 1 January 2024 that link another repo in the same ecosystem; after availability, test-change, and Linux filters, 4,437 groups remain. Manual review plus executable validation yield 192 eligible tasks, each with at least one fail-to-pass (F2P) test per target. All 60 bug fixes are kept; 60 of 132 features are sampled. The release covers 41 ecosystems, 253 target repos, 2,815 F2P tests, and 22,139 pass-to-pass (P2P) regression tests.
80% of tasks are producer-consumer: one repo supplies an interface, protocol, or artifact, another consumes it. 20% are parallel propagation, such as language SDKs or native plus polyfill. Median gold patch: 13 files, 416 lines. Prompts reuse issue and PR wording, merge overlapping asks, and spell out edge cases without implementation recipes. Hidden tests start from test hunks in the gold patches. Reviewers revised 88 of 120 tasks: 87 relaxed helper names, fixed paths, or error-message strings the request never required; 16 dropped checks for extras the gold patch added that the issue never asked for. Revised suites still fail on Base and pass on Gold. A task succeeds only if every target repo passes all of its F2P and P2P tests.
Seven setups: Codex CLI v0.147.0 with GPT-5.6-sol (high); Claude Code v2.1.139 with GPT-5.6-sol, Claude Opus 5, Gemini 3.8 Flash, DeepSeek V4 Pro, Qwen 3.8 Max, and GLM 5.3. RQ3 holds Codex CLI plus GPT-5.6-sol and the prompt fixed on 89 tasks, comparing one joint run over the ecosystem workspace with a separate full-budget run per target repo. 31 tasks needed per-repo prompt edits and are dropped. The comparison matches per-run limits, not total budget.
Full-task success runs from 10.83% (Gemini 3.8 Flash) to 42.50% (Codex CLI with GPT-5.6-sol). Swap GPT-5.6-sol from Codex CLI onto Claude Code and success falls to 32.50% while mean API calls rise from 86.7 to 134.8. More calls, fewer solves.
| Setup | Task success | Repo success | Mean API |
| Codex + GPT-5.6-sol | 42.50% | 63.64% | 86.7 |
| Claude Code + Qwen 3.8 Max | 37.50% | 56.92% | 177.1 |
| Claude Code + Opus 5 | 35.00% | 55.34% | 160.2 |
| Claude Code + GPT-5.6-sol | 32.50% | 56.92% | 134.8 |
| Claude Code + DeepSeek V4 Pro | 26.67% | 47.83% | 160.2 |
| Claude Code + GLM 5.3 | 20.00% | 39.92% | 151.1 |
| Claude Code + Gemini 3.8 Flash | 10.83% | 15.81% | 146.1 |
Codex fully solves at least one target repo on 83.33% of tasks, yet only 42.50% of whole tasks. In 48 of 69 failures, some repos pass every F2P test and others do not. Failures split into incomplete scope, recognized work never delivered, and post-edit misses. For Codex those shares are 37.68%, 2.90%, and 59.42%. Gemini's delivery failures are 52.34%; 82.14% of those stay in investigation or planning. Its feature success is 0.00%. Test-level numbers are kinder: Codex passes 81.74% of F2P tests and 98.84% of P2P tests. Old behavior mostly holds. New behavior often does not.
On Sentry's three SDKs, Codex narrows a shared strict trace-continuation policy to Go and leaves Python and Ruby untouched. On Godot/Native it diagnoses the fault at the Godot call site, implements only the Native API, watches 304 local unit tests pass, and still scores 0/1 F2P on Godot.
Qwen leads bug fixes at 48.33% versus Codex at 41.67%, then drops to 26.67% on features. All setups do worse on the 13 three-repo tasks than on the 107 two-repo ones. Codex repo success rises from 63.08% (two repos) to 66.67% (three), while task success falls from 42.99% to 38.46%. Among its failed three-repo tasks, 87.50% edit every target and 62.50% solve two of them. Coverage is not the bottleneck. Joint correctness is.
On 89 paired tasks, joint execution solves 40.45% and independent-all solves 35.96%. Independent helps bug fixes (44.83% vs 34.48%) and hurts features (31.67% vs 43.33%). 22.47% of tasks flip. Independent burns 296.6 API calls per task against 94.7 for joint (3.13 times) and still loses overall. Among repos that fail jointly, 60.00% of untouched ones recover in a solo run; only 9.43% of already-edited ones do.
This is a different evaluation from SWE-bench with extra languages. BeyondSWE's CrossRepo setting uses foreign repos as hints while scoring one target. WideSWE scores every repo that must change. SDK matrices and protocol endpoints look like this in production.
Repo-level pass rates overstate cross-repo skill: Codex's 63.64% repo success versus 42.50% task success is unfinished work next door. The scaffold gap is 10 points on the same model. Splitting work recovers omitted repos; it rarely repairs a wrong implementation already attempted. Feature work still needs the other repo as a behavioral reference. The useful next experiment is multi-repo orchestration and cross-repo checks.
There is no standalone limitations section. RQ3 gives each target a full run, so it is not an equal-budget test of splitting. The opposite bugfix and feature deltas are not to be generalized. Thirteen three-repo tasks is a thin slice.
Contamination is untested. Tasks require a latest related PR on or after 1 June 2025; the preprint is dated September 2026. Whether GPT-5.6, Opus 5, or Gemini 3.8 saw those merged PRs is unknown. Sampling starts from the star-count Top 200 and Linux-runnable repos. Features were hand-picked (60 of 132) toward runtime, protocol, schema, and SDK work.
Failure labels use final diffs plus trajectories. Two annotators agreed on 91.2% of unresolved runs (Cohen's κ=0.86). Test review touched 88 tasks in the direction of looser implementation constraints, which can let lucky patches through. The ethics note says agent patches can ship vulnerabilities and regressions and should not go out without review.