SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness
Haozhe Liu, Tian Ye, Sensen Gao, Qihang Cao, Yitong Li, Mingchen Zhuge, Duomin Wang, Ruihua Zhang, Ping Luo, Jiawang Bian, Lei Zhu, Ligeng Zhu, Enze Xie, Song Han
cs.AI
2026-09-17
Search over 500+ environments yields four Pi harness mechanisms. Versus Pi, SoL-Pi cuts tokens 44.7-49.0% and API cost about one third, with scores still close.
Once coding agents run long, the bill is trajectory tokens, not a single forward pass. Kernels, quantization, and cheaper models cut cost per token. The harness (how actions fire, when context is compacted, how huge observations are handled) is a second axis, and a local win can move failures and spend downstream. Humans staring at traces do not scale across environments. Prior automated harness search often looks good on search tasks and overfits held-out ones.
NVIDIA, NTU, and MIT put recursive self-improvement at the harness layer: a research agent reads baseline traces, proposes changes, and passes capability and efficiency gates in development environments, then freezes before held-out benchmarks. The target is reusable token savings, not a patch for one task.
An outer loop covers about 152 directions (context, progress, tools, delegation, prompt/policy, improvement/evaluation). Inner lineages implement, review, and harden independently. Search uses 535 executable environments: 495 GitHub issue–PR repos plus 40 synthetic tasks with executable verifiers, more than 3,000 runs and 60,000 interactions. Capability metrics and tolerances are fixed up front and out of the optimizer's reach. EdgeBench's 51 public tasks stay out of search: 11 for one-way acceptance of a frozen candidate, 40 for generalization, with no feedback into search.
Four surviving mechanisms land on Pi:
On public EdgeBench, versus Pi with GPT-5.6 Sol, the four-mechanism Efficiency stack scores 42.003 vs 44.833 (93.7% retained), tokens 1.0990B vs 2.1538B (−49.0%), cost $894 vs $1,339 (−33.2%). ObservationPack alone (Performance) raises the score to 47.208 with a 6.1% token cut. The same Efficiency stack transfers to Opus 5 with no extra search: 42.224 vs Pi 44.756 (94.3%), tokens −44.7%, cost −33.5%. Versus native Codex and Claude Code, API cost falls about 50.0% and 54.3%. The paper's hourly estimate is $8.75–$13.50 versus native harnesses and $4.36–$5.71 versus Pi.
On 63 CPU-only Terminal-Bench 4 tasks, SoL-Pi solves 15 against 18 for Codex and Pi, at $211.12 versus Pi's $286.45. On six IMO 2026 problems formalized in Lean 4, SoL-Pi passes three at $20.90 per pass, below Codex $22.89 and Pi $25.32. In a kernel-optimization swarm, 20 SoL-Pi workers reach 1,127 cycles at $60.11 in two hours, versus a Pi swarm at 1,366 / $82.12; a single agent remains cheapest ($39.20 / 1,333).
Every single mechanism reduces tokens. ObservationPack leads score on GPT-5.6 Sol, Action Fusion on Opus 5. The full stack cuts cache-read from 2.13B to 1.06B and raises cache-write from 0.014B to 0.032B, yet total cost still falls: breaking a cached prefix can be cheaper than resending a long context.
Without touching weights, harness search can cut the long-horizon bill and still run on a second backend. The four mechanisms hit round trips, compaction timing, observation replay, and log delegation, orthogonal to training a smaller model. Overnight exploration jobs can take the full stack; score-sensitive runs can keep ObservationPack or Action Fusion alone.
Search used only GPT-5.6 Sol traces, so Opus 5 triggers less often and less intensely. Eleven of 51 EdgeBench tasks participate in frozen acceptance; the strict held-out slice is 40. Terminal-Bench 4 solve count is below Pi (15 vs 18), so savings come with a small capability cut. Each swarm configuration is one two-hour run. The authors call cross-model transfer preliminary and do not report a breadth/depth scaling law. Using a cheaper harness to search the next one remains a stated vision, not a compounding result in this paper.