AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces
Sungho Park, Wonjoong Kim, Rongyuan Tan, Jue Zhang, Wook-Shin Han, Pengfei Gao, Chanyoung Park, Yongqiang Yao, Rao Fu, Elsie Nallipogu, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang
cs.AI, cs.CL, cs.LG, cs.MA, cs.SE
2026-08-24
Microsoft's AutoSaddler treats the agent harness as code and updates it offline from failure traces. It raises GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0 by 9.0, 9.6, and 10.0 points over the matching base harnesses.
Long-horizon agents fail by compounding small mistakes across many tool steps. The usual fix is an external harness: system prompts, tool interfaces, hooks, and loop logic. A good harness can raise success a lot, but the design space is large and each trial costs a full rollout.
Prompt-only search cannot touch tools or middleware. Unconstrained code edits tend to hot-fix the trajectory that just failed. Teams then rebuild the same layer when the backbone or domain changes.
AutoSaddler treats harness tuning as offline mini-batch learning. The search space is prompts, tools, and middleware. Memory and skill libraries are out of scope because tasks are assumed independent.
Each iteration evaluates the current harness on a training mini-batch. A Diagnosis-Patch agent, built on the Claude Agent SDK, inspects failed traces and harness source rather than summarizing the trace in one shot. Patches are typed: prompt rules, new tools, argument or implementation fixes, tool docstrings, PreToolUse hooks, infrastructure, and loop logic. Capability patches change executable code; Steering patches change text only. Optimization starts in a Capability phase and later switches to Steering.
A patch that improves the mini-batch is checked on a development set. A Reflection session then labels cases as fixed, regressed, still-failing, or still-passing and writes those lessons into EvoDAG, a DAG of prior harnesses. An Evolution session can recombine patches across lineages. When the rollout budget is spent, the best development-set candidate is tested once.
The default backbone is Claude Opus 4.6. Base harnesses are GAIA2's ReAct agent, SWE-agent, and Terminus 2. Automatic baselines are GEPA and Meta-Harness. Splits keep task groups disjoint; SWE-Bench Pro trains on qutebrowser and tests on Ansible, Flipt, and Element-web.
| Benchmark | Manual base | Best auto baseline | AutoSaddler |
| GAIA2 Pass@1 | 53.0% | GEPA 54.6% | 62.0% |
| SWE-Bench Pro | 37.3% | GEPA 42.5% | 46.9% |
| Terminal-Bench 2.0 | 40.0% | Meta-Harness 43.3% | 50.0% |
On Terminal-Bench it also beats the hand-tuned Terminus KIRA harness at 47.5%. A second GAIA2 run reaches 58.6%; training on a different universe reaches 57.4%. Transferring an Opus-optimized harness to Haiku 4.5 still adds 5.6 points over the base harness.
On the GAIA2 dev set, AutoSaddler hits 72.3% after about 1,000 task executions. GEPA and Meta-Harness stall at 64.6% and 61.5% after about 2,800. Counted as traces used for learning, the peak arrives at 147 traces, about 10 times fewer than Meta-Harness at 1,400.
Removing generalization-aware selection is the largest ablation: GAIA2 falls from 62.0% to 50.6%, below the manual base. Dropping in-depth diagnosis yields 57.8%; dropping structured intervention yields 56.9%. Without structure, 91.5% of patches become Steering text edits. New-tool, loop, and infra patches accept at 83%, 71%, and 67% on the mini-batch, and the full system raises their share above 25%. Capability patches fix at 55% versus 58% for Steering, with fewer regressions (8% vs 17%).
Most agent reliability work is still prompt and tool editing. This paper turns that craft into a loop: debug traces against code, constrain the edit types, and keep only patches that survive a held-out set. For an internal agent with a real eval, that is cheaper than retraining the backbone.
It is offline. It does not train weights and does not claim continual self-evolution.
The optimizer and the task agent both use Opus 4.6, so compute is not cheap. The paper notes higher optimizer cost per patch and relies on evaluating fewer trajectories. GEPA still leads on Element-web (45.2% vs 43.5%). Tasks are treated as stateless, so memory and skills are not searched. Optimization is one run and test evals are three repeats. The body text writes +8.4 points on SWE-Bench Pro, but 46.9 minus 37.3 is 9.6, matching the abstract and the table.