AgentBug-Smith auto-reproduces 200 real agent-harness bugs; SOTA repair agents fix just 9%

AgentBug-Smith: Automatically Reproducing Real-World Harness Bugs in Agentic Systems

Yiming Cheng, Alfin Wijaya Rahardja, Mengshi Zhang, Zihao Chen, Zhenpeng Chen, Yiling Lou

The University of Chicago / Fudan University / TensorBlock, Inc / Tsinghua University / University of Illinois Urbana-Champaign

cs.SE, cs.AI

2026-09-29

A multi-agent pipeline auto-reproduces real harness bugs from open-source agent repos at up to 28% success, building a 200-bug live benchmark where SOTA repair agents fix only 9%.

What problem this solves

An agentic system is a backbone LLM wrapped in a harness: the software layer handling context management, tool integration, orchestration, and guardrails. Harnesses have grown large (OpenClaw ships about 200K lines), and bugs in this layer are now a meaningful source of agent failures. They are not ordinary software defects. They live in agent-specific structures, involve constant interaction with model providers, tools, and network services, and often surface only in particular execution states.

The measured numbers are rough: software repair agents fix harness bugs at 4.67% versus 40.67% on general software bugs. Progress is bottlenecked by data. The only executable benchmark, AgentIssue-Bench, holds 43 bugs and cost 150 manual hours, with a fixed size that invites training-data contamination. Automated reproduction pipelines such as SWE-Factory and SWE-bench-Live were built for general software and mostly break on agent repositories.

Method

AgentBug-Smith is a fully automated multi-agent pipeline with three stages.

• Harness bug identification. Dual-stream repository discovery seeds the pool from community awesome lists, then consumes hourly GH Archive events to catch emerging projects. Repositories need at least 50 stars and tests, and an LLM judge verifies the codebase actually implements an agentic system. Issue mining keeps only closed issues linked to a merged PR, collapses the many-to-many relationships into one-to-one pairs so each bug has unique buggy and patched commits, and an LLM judge applies a six-category harness taxonomy to drop plain utility bugs.
• Environment construction. Rather than asking an LLM to write a Dockerfile from scratch, rules first parse dependency manifests and CI configs to infer the runtime, a single LLM call produces the initial image, and a tool-using agent iterates on build errors. Smoke testing samples about 20 existing tests, and assertion failures are ignored so refinement focuses on import and dependency problems. The key design: mock service environment variables injected at the container level redirect external model-client calls to local test harnesses, with no changes to the agent's source code.
• Failure-reproducing test generation. A test qualifies only if it fails on the buggy commit in the way the issue describes and passes on the patched commit. Mocking is allowed at model-client and network boundaries only; the function under test must run for real. Just 47.11% of issues ship an in-patch test, so the pipeline synthesizes the rest. When test-only refinement stalls, the test and the Dockerfile are co-optimized together.

Results

Reproduction success on 225 harness-bug issues, same pipeline across three backbone LLMs:

A 10.67 to 27.56 percentage-point lead at $0.56 to $2.44 per issue. Across all runs the pipeline reproduced 75 bugs, 45 of which neither baseline could reproduce, and manual inspection confirmed 74 of 75 (98.67%) as faithful reproductions. Identification holds up too: 95% accuracy on repositories and 92% on harness-bug issues, versus 35% and 46% for a vanilla LLM call.

The resulting Live-Harness-Bench holds 200 reproducible bugs: repositories average 123K lines of code, gold patches average 202.9 changed lines, 31.5% of bugs touch tool registries and action interfaces, 30.5% touch context and memory management, and the newest issues date to August 2026.

Three mainstream repair agents (all on GPT-4.1-mini) on the benchmark:

"Plausibly resolved" means the patch passes the reproducing tests; "correctly resolved" adds manual confirmation of semantic equivalence with the developer's patch. Against the 40.67% these agents report on SWE-Bench Verified, harness bugs are a different difficulty class. Used as a knowledge base (121 train, 79 test, no repository overlap), a distilled textual SKILL.md lifts mini-SWE-agent's correct resolution from 1.27% to 7.59%, with function-level localization rising from 34.60% to 66.41%.

Why it matters

This is the first executable harness-bug benchmark that grows with upstream development. The manual route cost 150 hours for 43 bugs; this pipeline pays a few dollars per issue, has already stacked up 200, and keeps ingesting new GitHub issues, pushing the contamination window forward.

The 9% versus 40.67% contrast settles a question: repair agents tuned on general software collapse on harness code. Claims that agents can fix agent infrastructure need this kind of benchmark, not a generic SWE score.

The skill-distillation experiment sketches a loop: real failures and fixes flow into the benchmark, and the benchmark feeds repair agents. The bigger it grows, the more the loop holds.

Limitations

Reproduction tops out at 28%. Unsuccessful runs number 162 to 180 per backbone, dominated by environment failures: dependency installation, Docker configuration, container disk exhaustion. The authors acknowledge that failure distributions shift across backbone models, with no general fix. Evidence for joint optimization is a single AgentScope #1297 trace, which the paper itself calls a representative execution record, not a causal ablation.

The downstream experiments are small. The distillation test set is 79 bugs, the improvement starts from a near-floor 1.27%, and 7.59% is still low in absolute terms, verified on mini-SWE-agent alone. Also, the baselines were never designed for harness bugs, so beating them shows general techniques do not transfer; how far the pipeline sits from practical use is answered by the 28% rate itself.

Terms

• Harness: the software layer around a backbone LLM handling context, tools, orchestration, and guardrails; it shapes agent behavior as much as the model does.
• In-patch test (IPT): a test submitted alongside the fix in the same PR, directly usable as a failure-reproducing test.
• Fail-to-pass (f2p): a test that fails before the patch and passes after it; the standard for a valid reproduction.
• SWE-Bench Verified: the human-validated subset of the standard GitHub-issue bug-fixing benchmark.
• GH Archive: hourly dumps of public GitHub events, usable to discover active repositories incrementally.

Source

• arXiv 2609.37864v1
• PDF
• Hugging Face

What people are saying

• AgentBug-Smith paper: top coding agents fix just 9% of agent harness bugs rohanpaul_ai

Related papers

• Adding call edges lifts function Recall@20 at least 27% over dense retrieval
• 6,840 CABRA tasks stay near-perfect for agents until code merging tests understanding
• TestPrism: frontier suites pass 59.67% of references, 28% of full panels
• Google's FlowAgent: 67% of 195 pre-submit fixes correct; 28,554 applied
• Best of six agents scores 42.86 on 101 Simulink engineering tasks
• TUA-Bench: Top Terminal Agents Hit 65.8%, and Swapping the Harness Flips Rankings

All paper explainers