Microsoft Introduces LoopsBench for Long-Horizon Coding Agents; Top System Solves Only 25% of Tasks

机器之心 · wechat · 2026-08-22

Microsoft, in collaboration with Nanjing University, has released LoopsBench, a benchmark designed to evaluate Coding Agents on long-horizon software engineering tasks. Unlike traditional benchmarks like SWE-bench that focus on one-off issue resolution, LoopsBench emphasizes continuous execution, task dependency management, and regression control.

Key Innovations:

Dependency DAG: Decomposes long-term tasks into multiple verifiable Development Units. It constructs a dependency graph based on real code calls or inheritance, evaluating the execution path rather than just the final outcome.
Flow-aware Runtime: The evaluator only tests tasks whose dependencies are met (Ready Frontier). Completed units become Regression Obligations, forcing the agent to preserve existing functionality during future development.
Data Source: Comprises 112 tasks with over 5,300 development units, sourced from university course labs, consecutive GitHub PR sequences, and research code evolution.

Experimental Findings:

Even the best configuration achieved a Task Resolve Rate of only 25% and a Test Pass Rate of 53.05%.
Introducing an outer Continuation Loop improved the resolve rate from 17% to 25%, but it does not solve issues related to deep dependency progression.
Current agents often fail to fully recover task dependencies in their planning, either compressing parallel tasks into a linear chain or prematurely parallelizing tasks that should be sequential.

Original post →

Microsoft Introduces LoopsBench for Long-Horizon Coding Agents; Top System Solves Only 25% of Tasks

More from coding & agent