Microsoft Introduces LoopsBench for Long-Horizon Coding Agents; Top System Solves Only 25% of Tasks
机器之心 · wechat · 2026-08-22
Microsoft, in collaboration with Nanjing University, has released LoopsBench, a benchmark designed to evaluate Coding Agents on long-horizon software engineering tasks. Unlike traditional benchmarks like SWE-bench that focus on one-off issue resolution, LoopsBench emphasizes continuous execution, task dependency management, and regression control.
Key Innovations:
- Dependency DAG: Decomposes long-term tasks into multiple verifiable Development Units. It constructs a dependency graph based on real code calls or inheritance, evaluating the execution path rather than just the final outcome.
- Flow-aware Runtime: The evaluator only tests tasks whose dependencies are met (Ready Frontier). Completed units become Regression Obligations, forcing the agent to preserve existing functionality during future development.
- Data Source: Comprises 112 tasks with over 5,300 development units, sourced from university course labs, consecutive GitHub PR sequences, and research code evolution.
Experimental Findings:
- Even the best configuration achieved a Task Resolve Rate of only 25% and a Test Pass Rate of 53.05%.
- Introducing an outer Continuation Loop improved the resolve rate from 17% to 25%, but it does not solve issues related to deep dependency progression.
- Current agents often fail to fully recover task dependencies in their planning, either compressing parallel tasks into a linear chain or prematurely parallelizing tasks that should be sequential.
More from coding & agent
- Veracode: AI now writes ~half of committed code, but security pass rate stuck at 56% — WeldPond · 2026-10-16
- Engineer: my job went from silently thinking at a computer to constantly talking to it — KevinNaughtonJr · 2026-10-16
- LangChain launches Managed Deep Agents: build production agents as a directory of files — hwchase17 · 2026-10-16
- Vercel Goes Full Backend: Python GA, Containers, and Daemons — cramforce · 2026-10-16
- WorldMemArena Accepted at NeurIPS: Memory Storage ≠ Memory Use — xwang_lk · 2026-10-16
- Vercel Ship Demos Container Deploys for Go, Ruby, Java and More — edwin · 2026-10-16