Harness Learning Enables Generalizable Test-Time Adaptation
Alvin Zhang, Xuecheng Liu, Zixuan Wang, Fahim Tajwar, Daman Arora, Ruslan Salakhutdinov, Daniel Khashabi, Yuda Song, Andrea Zanette
cs.CL, cs.LG
2026-09-29
A 4B RL proposer revises harnesses from execution feedback. On 21 unseen families, mean score rises 0.32 to 0.62, beating a 35B teacher; HotpotQA training transfers to MuSiQue.
A language-model agent is a model plus a harness, the executable program that schedules calls, tools, and information flow. Freeze the weights, swap the harness, and scores can jump. Current pipelines still hunt for those edits at test time by hand or by search, and a revision that works on one task often dies on the next.
What is missing is a revision procedure that transfers. Train it once, then keep both models frozen and let execution feedback drive code edits on new tasks.
Harness learning is meta-learning over executable programs. An outer loop trains a proposer. An inner loop uses that proposer to adapt a harness from execution feedback. Harness edits stand in for weight updates: they change control flow and tool interfaces while parameters stay fixed.
The proposer reads a task description, the current harness, and an execution report, then writes a structured code edit that names a span and its replacement. Apply parses the patch and marks it invalid when parsing fails. Only the proposer is trained. The solver stays frozen.
Reward is mean grader score of the revised harness on a scoring set, plus small bonuses for a parseable, runnable edit. Scoring questions are disjoint from the questions that built the execution report. Training uses GRPO (group relative policy optimization), which normalizes advantages within a group of edits for the same context.
Two recipes. On Reasoning Gym, proposer and solver are both Qwen3.5-4B. Successful revisions from a Qwen3.6-35B-A3B teacher initialize SFT (432 examples, about 48% interpreter loops that have the solver emit code for a runtime). GRPO follows. On multi-hop QA, a Qwen3-4B proposer and Qwen3-8B solver start RL from the base model with no teacher demos, trained only on HotpotQA.
At test time the same proposer can run for several rounds. Reasoning Gym samples one candidate per round and keeps it only if development score rises. QA samples eight and advances the best even when it scores below its parent.
Single-step revision on 21 Reasoning Gym families held out of both SFT and RL, mean over eight proposals:
| Method | Mean | best-of-8 |
| Base (untrained 4B) | 0.32 | 0.63 |
| SFT | 0.49 | 0.78 |
| Single-step RL (4B) | 0.62 | 0.83 |
| Teacher (35B) | 0.56 | 0.85 |
After RL, the average proposal already beats the seed harness. Base and SFT averages still sit below it. The 35B teacher still wins under oracle best-of-eight selection, which picks the best held-out score after the fact and is not available at test time. On unseen families, failed or zero-scoring proposals drop from 38% (Base) to 8% (RL). SFT barely moves, at 35%.
Over five sequential rounds, Single-step RL takes most of its format-varied gain in round one; SFT climbs more slowly. After five rounds on 24 families, format-varied scores reach 0.725 for Single-step RL versus 0.498 for Base. On canonical questions the three trained proposers cluster near 0.70. Five sequential edits reach or pass the oracle best of eight independent proposals.
A proposer trained on HotpotQA transfers to MuSiQue and 2WikiMultihopQA. On MuSiQue, independent revisions average 0.15 exact match, only slightly above the seed; ten sequential rounds reach 0.27. The first round supplies one third to two thirds of that ten-round gain. On both unseen benchmarks, the mean final score of four ten-round runs beats the oracle best of 80 independent proposals. Proposal count per run is matched. Total compute and feedback are not.
Training on revision sequences does not stably beat training on single revisions. Multistep RL finishes slightly higher on canonical Reasoning Gym questions and lower on format-varied ones (0.649 vs 0.725). QA finals are nearly identical. A policy trained only on single-step contexts already improves harnesses across later rounds.
The harnesses themselves change shape. On Reasoning Gym, SFT and RL mostly emit interpreter loops. On QA, Base sometimes collapses the workflow to a single solver call or removes the solver. RL keeps multi-hop retrieval. A typical edit still uses summaries to form the next query, then feeds retrieved passages straight into the answering call.
For people building agents, test-time adaptation moves from weights to programs. A frozen 4B proposer, reading feedback on a new task, produces edits whose average score beats the 35B teacher that supervised it. SFT copies the teacher's interpreter-loop habit. The RL stage is what pushes mean proposal quality past the teacher.
The setting is narrow: a programmable grader, development-set feedback, and a Python harness that can take patches. This is not a general agent training recipe. The direction is sharp. Experience lands in a revision policy. The solver weights never move.
The paper is clear about the scaffolding: prescribed revision prompts, a frozen solver, and Reasoning Gym SFT from a single teacher that almost only demonstrates first-round edits. Composition bonuses and helper-tool directives do not help consistently. When a generated helper is algorithmically right, parsing, tool wiring, or answer extraction still often break the chain.
The main figures also drop five unseen families that neither the 4B proposers nor the teacher could handle with an algorithmic harness. Across all 26 unseen families, Single-step RL falls from 0.62 to 0.528 and the teacher to 0.501. The gap survives, thinner. The QA sequential-versus-independent comparison matches 80 proposals per run and leaves compute and feedback unmatched. Multistep training still assigns credit one revision at a time; credit across a chain is untested. Seed harnesses differ sharply between the two settings, so how far multi-round gains can go is still mixed up with how rich the starting program already was.