ActiveSaddler picks which failures to train on, adding up to 7.5 points to harness optimization

ActiveSaddler: Automated Curriculum Learning for Agent Harness Optimization

Sungho Park, Wonjoong Kim, Jue Zhang, Wook-Shin Han, Pengfei Gao, Chanyoung Park, Yongqiang Yao, Rao Fu, Elsie Nallipogu, Qingwei Lin, Victor Rühle

cs.AI, cs.CL, cs.LG, cs.MA, cs.SE

2026-10-01

ActiveSaddler turns scenario selection for harness optimization into a bandit over failure patterns, adding 4.4/7.5 Pass@1 points on GAIA2 and Terminal-Bench 2.0 at equal budget.

What problem this solves

How reliably an LLM agent completes multi-step work depends heavily on its harness: the prompts, tool interfaces, and runtime control logic wrapped around the model. Automated harness optimization (AHO) methods such as GEPA, Meta-Harness, and AutoSaddler execute the current harness on training scenarios, diagnose weaknesses from traces, patch, and iterate.

All of them optimize how the harness is updated. Which scenarios generate the feedback is fixed before optimization starts: the full training set or a pre-scheduled mini-batch order. But the harness keeps changing, and so does the value of each scenario. An unresolved failure deserves another repair round; a repaired one has nothing left to teach. New failure modes sit undiscovered in unseen scenarios while old ones lose their remaining value. A fixed order handles neither side: many failures observed during fixed-order training remain unresolved in the final harness (Figure 3b). Under a limited rollout budget, that waste hurts.

Method

ActiveSaddler casts the choice of training scenarios as a non-stationary multi-armed bandit. The arms are neither task categories nor individual scenarios but failure patterns: diagnosed, plausibly repairable harness weaknesses that recur across scenarios and traces. The curriculum layer wraps a fixed optimizer (AutoSaddler in the main experiments) and never touches its update mechanism.

• Failure-Pattern Extractor: after each optimization step, every failure is first abstracted into a symptom independently (avoiding anchoring on existing arms), then normalized against the arm set: matched, composed, or registered as new. Each arm carries a description, supporting scenarios, and evidence; when selected, its scenarios form the next mini-batch (at most three per iteration).
• Arm Prioritizer: an LLM scores each arm in [0,1] on severity (still occurring?), fixability, breadth of generalization, and side-effect risk of a patch. Scores are recomputed every iteration, and arms are sampled from a softmax with temperature 0.15, so low-scored arms stay reachable as their utility shifts.
• Exploration Controller: an LLM makes a binary call each iteration. DRAW samples unseen scenarios to discover new failure modes; PULL revisits a known arm. An empty arm set forces DRAW; an empty unseen pool forces PULL. A regression-rediscovery rule re-admits previously successful scenarios once everything has been explored and every arm revisited, so lost repairs resurface without full regression re-runs.

Why failure patterns rather than categories or scenarios? The ablations answer directly. Category arms are too coarse: in one GAIA2 case, a still-failing scenario was folded into a broad time-category arm; after sibling scenarios passed, the arm's severity fell from 0.62 to 0.12 and that scenario was never sampled again. Scenario arms are too fine: a single weakness fragments across up to 75 arms on GAIA2, more than twice the maximum of 35 failure-pattern arms, so failures rarely get revisited under a limited budget.

Results

Both benchmarks use gpt-5.5 for the task agent and the optimizer (medium and xhigh reasoning), a shared rollout budget (1,400 rollouts on GAIA2, 490 on TB2), and the mean of three test-time executions:

Same optimizer, same budget: +4.4 and +7.5 points. Against difficulty-ordered curricula the gap reaches +9.2 points (TB2), and 80.0 beats the hand-engineered Terminus-KIRA (69.2) by 10.8 points.

All three components matter: swapping in category arms costs 3.0/10.8 points (GAIA2/TB2), scenario arms 3.6/6.7, removing the Arm Prioritizer 4.5/7.5, removing the Exploration Controller 4.5/8.3. Non-LLM substitutes also fall short on GAIA2: EMA failure-persistence scoring reaches 56.7% and count-based UCB-AIR exploration 55.6%, versus 59.8%.

The mechanism checks out. Failure-pattern arms aim iterations at still-failing scenarios more often than category and scenario arms by 16.5/35.3 points on GAIA2 and 29.9/23.8 on TB2; of failures seen during training, 75.0% are fixed by the final harness on GAIA2 versus 50.0% for AutoSaddler. The DRAW rate falls from 44% in the first half of optimization to 20% in the second, and unresolved arms are 2.0x (GAIA2) and 6.9x (TB2) more numerous at PULL than at DRAW decisions, while a fixed schedule sits near 1x, insensitive to state.

It transfers: layered onto GEPA, GAIA2 rises from 54.2% to 57.2%; a second independent run gives 61.6%, still above every fixed-order baseline. The curriculum layer raises optimizer-side cost per patch from $7.87 to $11.44 on GAIA2, but end-to-end it is cheaper: reaching 58.5% dev accuracy costs $298 versus $1,360 for AutoSaddler; on TB2, 78.9% at $128 versus $220.

Why it matters

Teams optimizing agents have focused on how to update: reflective evolution in GEPA, diagnose-and-patch in AutoSaddler. This paper shows that which scenarios to train on is worth several points on its own, and it plugs into a different optimizer. What it saves is rollouts, the expensive part: spending them on unresolved weaknesses beats a uniform pass over the training set. The same idea should transfer to other feedback-driven improvement loops such as tool-definition or skill-library optimization.

To be clear, this is an incremental improvement: it replaces no optimizer, it makes existing ones earn more per rollout.

Limitations

The paper has no dedicated limitations section; the points below mix its own disclosures with gaps the experiments leave open.

• Everything runs on gpt-5.5; whether the LLM-based scoring and exploration decisions hold on smaller or different models is untested.
• The three curriculum decisions are LLM sessions with their own stochasticity. Two-run robustness is shown only on GAIA2; TB2 has a single run.
• TB2 has 89 tasks total and a 40-task test split, so 80.0 ± 2.5 comes with a wide interval.
• Optimizer-side cost per patch rises 45-68% ($6.77 to $11.37 on TB2); the saving comes from expensive task-agent rollouts. Where optimizer calls are cheap relative to executions, the accounting could flip.
• Code and the project website are listed as 'will be available' and were not released with the preprint.
• The authors disclose generative-AI assistance for writing and literature search.

Terms

• Harness: the scaffolding around an LLM agent; prompts, tool interfaces, and runtime control logic.
• AHO (automated harness optimization): iteratively patching a harness from execution-trace feedback.
• Failure-pattern arm: a recurring, repairable harness weakness grouped across failed traces, treated as one bandit arm.
• Non-stationary bandit: a multi-armed bandit whose arm set and payoffs change over time; arms are added online here.
• DRAW / PULL: the two curriculum actions: sample unseen scenarios to discover failures, or revisit a known weakness.
• Pass@1: pass rate of a single execution, averaged over three test-time runs here.

Source

• arXiv 2610.00906v1
• PDF
• Hugging Face

Related papers

• Sharpening Tax: RL post-training lifts agent pass@1 while shrinking pass@K below the base model
• Explicit belief states take first place on all four agent benchmarks, at 5x token cost
• RASO mines public agent skills as priors, beating TextGrad and GEPA on all four benchmarks
• Agents' Last Exam: real job tasks from 250+ experts; top agent scores 0% on hardest tier
• 1 in 5 U.S. adults says AI is already sentient; 69% back a ban on building sentient AI
• Google's Cogentic solved five open math problems with a multi-agent Gemini harness in O(100) calls

All paper explainers