LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluation
Han Li, Zhemin Fang, Rili Feng, Yingqi Zhao, Jiaheng Liu, Pengfei Gao, He Ye, Dayi Lin, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang
cs.SE, cs.CL
2026-08-01
LoopsBench turns 112 real coding tasks into dependency DAGs. Opus-4.7 with Claude Code and outer continuation resolves 25%; plans miss prerequisites and regressions persist.
Coding agents have moved from single-issue patches to software work that runs for a long time. The infrastructure moved with them. Claude Code and Codex goal modes, plus dynamic workflows, add a loop on top of the harness so objectives, progress checks, and work assignment survive a long run.
Most benchmarks still score a terminal state. SWE-bench-style tasks hand over a self-contained issue; feature-level suites enlarge the change; the grade is still whether the final tests pass. They do not show whether intermediate obligations were kept, whether later edits regressed earlier units, or whether the agent followed a viable order through dependent work.
LoopsBench, from Microsoft, Nanjing University, UCL, and Shanghai Jiao Tong, treats that gap as loop engineering rather than harness engineering.
Each task is a dependency DAG. Nodes are separately testable development units. Edges are source-evidenced prerequisites: sequential PR chains, module reuse, producer-consumer APIs, and compositional layering. Hot files and lockfiles are denylisted so they cannot fake a chain. The graph is a descriptive reference recovered from source history, not a claim of an optimal schedule.
The release has 112 tasks from three authentic pools: 29 PR sequences, 57 course labs, and 26 research evolutions, across 8 languages and 9 domains. Median dependency depth is 6, with more than 5,300 development units. Selection requires a span of at least 2.5 months and a source-specific scale of at least 1,200.
The runtime releases tests along the ready frontier: a successor scores only after all predecessors pass, and finished nodes stay live as regression obligations. Agents may edit in any order; they are not told which unit is currently ready. A dual-container snapshot pipeline separates editing from adjudication and records loop traces at real change points.
Task instructions are recovered offline from the gold diff with Claude Code, stripping implementation path, repo names, and paper titles. Gold solutions and the active frontier stay hidden.
The strongest setup is Opus-4.7 with Claude Code plus outer continuation: Resolve Rate 25.00%, test pass rate 53.05%, normalized dependency depth 0.61. Without the outer loop it drops to 16.96%. GPT-5.5 on Codex with continuation reaches 21.43%. Under a fixed Claude Code loop, GPT-5.5 hits 20.54%, and both GLM-5.1 and DeepSeek-V4P hit 18.75%. Open-source loops are weaker with continuation: OpenHands 9.82%, SWE-agent 8.93%, mini-swe-agent 7.14%.
Plans recover only part of the source DAG. Claude Code's edge F1 is 0.71 and layer correlation 0.65; mini-swe-agent falls to 0.27 and 0.22. Closed-source loops plan tree-shaped concurrency with width ratio slightly above 1; open-source loops collapse toward a chain. On units they do resolve, patches run 1.58x to 2.54x the gold length. Agent-authored tests stay sparse versus the native suite. Claude Code's regression rate is 7.11%. Lighter loops look cleaner on regression because they produce fewer passing obligations that can regress.
Among four loop profiles, Claude dynamic workflows use 97.96 context-budget rounds per run and resolve 24.11%; Ralph uses 13.24 rounds and resolves 7.84%. Dynamic workflows still log 0.36 regression events per run. Renewing context does not remove regression pressure.
Anyone shipping a long-horizon coding agent cannot stop at SWE-bench. This suite splits model choice from loop implementation: swap the loop under one model, scores move; swap the model under one loop, scores move. The remaining failure is global planning, residual routing, and keeping finished obligations intact.
The usable artifact is the trace: edge F1, width ratio, patch inflation, tests written, regression rate. Closed-source loops over-parallelize; open-source loops serialize; both under-test. Outer continuation reduces early stalls. It cannot rescue later units that inherit unfinished earlier obligations.
The recovered DAG is a lower bound. It can miss configuration, data formats, build systems, and cross-service behavior. The fixed graph is an evaluation contract; exploratory redesigns that replace earlier requirements sit outside it. The pool favors sources with auditable prerequisite evidence, so mobile, frontend-heavy, and hardware-adjacent projects are largely absent.
Correctness is tied to the released suite, not semantic equivalence. Hidden checkpoints mix limited feedback into the regression number. Construction used Claude Code and Opus-4.7, so wording, unit boundaries, and tests may carry model bias, and public sources leave contamination risk. Evaluation is expensive, and proprietary outputs can drift. LoopsBench measures one executable contract, not deployment readiness.