MemoHarness: Agent Harnesses That Learn from Experience
Yue Huang, Wenjie Wang, Han Bao, Yuchen Ma, Xiaonan Luo, Yi Nian, Haomin Zhuang, Zheyuan Liu, Yue Zhao, Xiangliang Zhang
cs.AI, cs.CL
2026-07-15
MemoHarness adapts a six-dimension harness from past runs with no test labels, hitting 0.806 on Terminal-Bench vs Codex 0.722, mean +0.098 on six other models.
Keep the base model fixed and change the agent harness around it, how context is packed, which tools fire, how decoding is set, how turns are orchestrated, what memory persists, how outputs are cleaned up, and task success can move by tens of points. Automatic optimization still edits narrower objects: prompts, DSPy-style pipelines, workflow graphs. Meta-Harness searches over harness code, then ships one training-time artifact for every case at deployment.
A single global harness breaks down once cases differ in domain, reasoning depth, retrieval need, or output format. MemoHarness searches for a strong global harness, then specializes it per test case with no labels, no feedback, and no extra search.
The harness is split into six editable control surfaces along the inference timeline.
Think of this as compiler flags for the same source, not a rewrite of the program. Search starts from a minimal harness: no demos, no tools, deterministic single-call decoding, no memory, raw passthrough. For ten outer iterations a controller proposes the next configuration from the experience bank and runs it on the labeled search set. Task reward ranks first; token usage breaks ties only.
The bank has two layers. Per-case entries store the configuration delta, trajectory, reward, cost, and a diagnosis that names a primary failure dimension. Global patterns are distilled from failure clusters. The diagnostic operator is coarse on purpose: verifier outcomes, timeouts, missing artifacts, and command failures are mapped onto one of the six dimensions. At test time the bank is frozen. Cosine similarity over instruction embeddings retrieves the nearest successes and failures, and the global harness is rewritten for that case. Easy items can stay light; retrieval-heavy or format-sensitive items pick up extra orchestration only when the retrieved evidence warrants it.
The headline number is on an 18-task held-out split of Terminal-Bench (80/20 of the 89-task suite) with GPT-5.3-Codex as the base model. MemoHarness reaches 0.806, +0.084 over the strongest fixed harness in the comparison, Codex at 0.722, and +0.250 to +0.445 over Claude Code, OpenCode, and Terminus. Product baselines are closest released configurations, not pure scaffold transplants, and the paper says so.
Validation-selected harnesses beat the minimal start on all three families.
| Task | Start | Selected |
| Terminal-Bench | 0.722 | 0.806 |
| LiveCodeBench | 0.900 | 0.967 |
| FinanceAgent | 0.600 | 0.767 |
Search is not monotonic. Terminal-Bench hits 0.833 at iteration 4 on the held-out split; LiveCodeBench touches 1.000 at iteration 3. The shipped harness is chosen on validation, so it can sit below the training peak. FinanceAgent, the long-horizon analytical task, still moves on the search set, from 42.5% up to a 65.0% peak near iterations 8 and 9. LiveCodeBench is near the model ceiling and oscillates in a roughly 4-point band.
Transfer across unseen suites is selective. The Terminal-Bench harness lifts MMMLU 0.818 to 0.848, StrongReject 0.879 to 0.909, and SWE-Bench Pro 0.706 to 0.765. HumanEvalFix and Reasoning-Gym-Easy are already saturated and do not move. The LiveCodeBench harness pushes MMMLU to 0.879, then drops LawBench from 0.675 to 0.669.
The same GPT-5.3-Codex-searched harness, applied with no retraining, raises every one of six other models on Terminal-Bench, mean +0.098. GLM-5 goes 0.500 to 0.733. GPT-4.1 only goes 0.500 to 0.538. DeepSeek-V3.2 goes 0.333 to 0.444.
Dollar cost is computed after the fact from public list prices. MemoHarness reads 14.18M input tokens because it retrieves from the bank, but 13.32M of those are cached, so the 18-task split costs about $6.89, under Codex at $10.28 and Claude Code at $9.51. OpenCode is cheaper at $2.34, with much lower accuracy. If cache reuse falls, this comparison changes.
A lot of agent work is now harness work. MemoHarness turns that layer from handwritten defaults into something searchable, diagnostic, and case-adaptive. Gains are larger on long-horizon, tool-heavy tasks than on near-saturated single-shot coding. Cross-model lifts suggest the learned policy is not only a prompt quirk of one generator.
This is still incremental. Search needs labels. Test-time adaptation does not loop. The paper never fully ablates the bank, the global patterns, and the per-case rewrite. The practical setting is closer to "you have verifier-backed run logs and want a better shell for similar tasks" than unsupervised online self-improvement.
The appendix is blunt. The headline scores are point estimates on 18 held-out tasks, with no confidence intervals and no significance tests. Not every baseline shares a model and runtime. Component attribution is missing. Cost depends on cache hits. The controller and diagnostic operators are heuristics.
Two further weak spots. The text claims +0.250 to +0.445 over the remaining baselines without printing Terminus, OpenCode, and Claude Code success rates in the body, so those gaps live in a figure. The GLM-5 jump of 0.233 is large, and the paper does not show how weak that model's default harness was, so some of the gain may be a low starting point.