BOXBOX: agents pit-stop mid-run to rewrite and hot-reload their harness, hitting 1.70x Speedup@4h vs 1.54x

BOXBOX: Recursive Harness Self-Improvement for Long-Running Performance Optimization

Yuhao Tan, Hanyun Jiang, Fangkai Yang, Yuan Yao, Taolue Chen, Bo Qiao, Yu Kang, Lu Wang, Pu Zhao, Xiaoxing Ma, Qingwei Lin

cs.SE, cs.AI

2026-11-06

Agents trigger, revise, and hot-reload their own harness mid-task; BOXBOX reaches 1.70x Speedup@4h on the new PerfRace benchmark, ahead of 1.54x for the strongest baseline.

What problem this solves

Everything an LLM agent can do passes through its harness: the executable scaffolding of prompts, tools, and control logic that defines which actions the model may take and what it gets to observe. Keep the model, swap the harness, and measured performance moves. Automated harness optimization grew out of that observation: an optimizer reads trajectories and evaluation feedback, rewrites the harness, and ships the result for reuse.

Current methods do this offline, diagnosing failures on training tasks with reference answers and iterating toward a 'general' harness. Real tasks are open-ended and run for hours. There is no clean failure signal to diagnose against, and long runs keep surfacing difficulties that revisions learned on other tasks never anticipated. The paper's own baseline shows the failure mode: offline Harness Evolution bottoms out at 1.281x over four hours on their new benchmark, below the 1.442x of a harness left completely untouched.

Test-time alternatives exist but run an outer loop: an external controller schedules revisions between attempts. The agent can see itself stuck, yet has no say in whether to spend its remaining budget on the work or on the tooling, and every fix waits for the next attempt. This paper, from Microsoft with Nanjing University and others, targets exactly that gap.

Method

The paper contributes a benchmark and a system.

PerfRace. The team mined 103,475 candidate projects from public GitHub coding-agent activity, filtered them with deterministic prechecks down to 7,257, used GPT-5.6-Luna to assess optimization potential down to 239, then finished with agent-human pair verification to produce 100 tasks, at an average of 9.1 hours of curation and 12 human-agent interactions per task. Tasks span seven domains and sixteen languages. Each gives the agent a real project and a fixed end-to-end workload (say, compiling SQLite with the CCC compiler), with no reference patch: the agent profiles, finds bottlenecks, edits code, and submits whenever it wants within four hours. The score is the best-so-far correctness-preserving speedup. Shadow verification runs scoring in separate containers, keeping verifier code and held-out tests out of the agent's workspace.

BOXBOX, named after the F1 'box, box' pit-stop call, hands the revision decision to the task agent and backs it with kernel-managed hot reloading:

Revisions are chosen by expected benefit under the remaining budget, for instance folding a recurring tool-call sequence into one reusable tool.

Results

Fifty held-out test tasks, three four-hour runs per method, all with GPT-5.6-Luna and the same starting harness (Terminus-KIRA):

MethodSpeedup@4hModel cost/runSpeedup/$
Frozen Harness1.442x$23.220.2142
Harness Evolution (offline)1.281x$24.560.1243
Harness Scaling (outer loop)1.539x$12.980.2872
BOXBOX1.702x$16.430.4886

The details matter as much as the headline:

Why it matters

For agent infrastructure work, this paper turns harness optimization from an offline batch step into an in-run capability, and the evidence says the agent's own timing judgment beats both timers and external controllers. Hot reloading with preserved context is the key engineering contribution: session resets like HS's save tokens but discard hard-won experimental findings; BOXBOX keeps both, and its speedup per dollar (0.4886) is roughly 2.3x Frozen's.

PerfRace deserves attention on its own. Existing code-optimization benchmarks (PIE, GSO, SWE-Perf) build tasks around known performance patches; PerfRace hands over an end-to-end workload with no reference answer, closer to real performance engineering, and its pipeline keeps mining fresh tasks from GitHub.

Called honestly, 1.70x versus 1.54x is an incremental gain on a single benchmark, not a step change, though each design claim (agent control, full editing scope, context preservation) has an ablation behind it.

Limitations

The paper devotes a section to security: letting an agent modify executable harness code mid-task widens the risk surface beyond a fixed harness, potentially introducing security regressions or weakening safeguards, and self-modifying mechanisms could be misused to bypass operator controls. Kernel validation plus rollback protects the experiment, not production; the authors position BOXBOX as a research prototype requiring sandboxing, access control, monitoring, and human review before any deployment.

Gaps the paper does not cover: every experiment runs on a single model (GPT-5.6-Luna), so whether cheaper models should still budget for harness revision is untested; ablations cover only 14 tasks; part of HE's failure traces to adaptation choices (training runs capped at 45 minutes), so the offline route's floor may be underestimated. And with reviews consuming just 2.1% of time yet yielding clear gains, the plausible story is that the payoff comes from a better optimization workflow, but the paper never separates the causal contribution of harness edits from the mere benefit of an extra reflection pause.

Terms

Source

What people are saying

Related papers

All paper explainers