Learning Meta-Skills for Agent Harness Design in Test-Time AI4AI
Cheng Qian, Kunlun Zhu, Beibin Li, Zhenhailong Wang, Heng Ji
cs.AI, cs.CL, cs.LG
2026-09-30
A Builder learns meta-skills from Target feedback and compiles them into per-task harnesses: 65.31% macro-average, +8.95 over no-skill building, +12.02 over direct delivery.
An agent's score reflects two things: how well the model reasons, and what environment it runs in. The same model solves a task when given proper tools and a verification loop, then fails it in a bare sandbox. Existing improvement paths either touch weights (RL, fine-tuning), search over agent programs and workflows at the system level, or distill execution traces into textual skills pasted back into the prompt. The last one has a structural weakness: the knowledge stays declarative, so at inference time the model still has to read the advice, remember it, and apply it correctly, every step of the way.
The paper's framing is an advisor and a PhD student. A capable student wastes days because configs and results are scattered; a good advisor does not run the experiments, just sets up a shared log, reproducible tools, and a validation workflow. Here a Builder model plays that advisor, learning from a Target model's execution feedback how to construct better environments, with both models' weights frozen throughout.
The central object is a meta-skill, a structured support principle with three fixed fields:
Learning runs on a development split as a build-execute-reflect loop. The Builder starts with an empty bank and constructs a harness for each dev task; the Target executes and leaves a public trace plus a benchmark score; the Builder reviews its harness and that feedback to add, revise, or keep the bank. Each batch allows at most one update, and it must cite evidence observed in that batch. After two passes the bank is frozen.
At test time the Builder receives the frozen bank whole or as BM25 top-2 retrieval and builds a fresh harness per task from seven component families: instructions, memory, context organization, composed tools, execution control, verification and recovery, and workspace preparation. The output is code-level machinery, such as file-version memory plus an audit-gated submission check, not a longer prompt.
Two design choices carry the weight. The triple structure forces each principle to state its trigger and responsibility boundary; a baseline that consolidates traces into free-form notes does clearly worse, which supports the structure. And handing skills to the Builder to compile moves the burden of interpreting advice from inference time to construction time, which is where the largest gains in the comparisons come from.
Builder fixed as GPT-5.6-Sol; Targets are Gemini-3.6-Flash, Qwen3.8-Flash, and GPT-OSS-120B. Harness-Bench covers agent workflows (95 test tasks) and NewtonBench interactive scientific law discovery (292), each reserving 10% of tasks for skill learning.
| Method | Macro avg. |
| Native environment, no Builder | 51.40% |
| Builder, no skills | 56.36% |
| Same skill bank delivered to Target | 53.29% |
| Independently learned structured Target skills | 54.38% |
| Builder meta-skills, BM25 top-2 | 60.45% |
| Builder meta-skills, full bank | 65.31% |
The takeaway for agent builders: compile experience into the environment instead of appending it to the prompt. The same knowledge is worth far more in the hands of a party that can turn it into tools, state, and verification logic than in the hands of the party that must read it mid-inference, and the 12.02-point gap prices exactly that difference. The everyday translation is to write the check as a script rather than another paragraph of system prompt. The same-model results (+18.71) also sketch a self-improvement path that touches no weights, orthogonal to RL and fine-tuning.
The paper draws its own boundaries: the evidence covers one main Builder, two benchmarks, and a single execution per task and condition; Builder-side construction cost sits outside the evaluated budget and is never weighed against the gains. Refinement is not monotonic, as Qwen peaks after one pass on Harness-Bench and then drops 4.06 points, meaning revisions can overfit recent evidence. Cross-Builder transfer gains all have confidence intervals containing zero, and the memory-plus-context ablation settles nothing.
Two more concerns from a close read. The dev splits hold only 11 and 32 tasks, and the paper never reports the final bank size or how often each skill fires, so the generalization boundary is unclear. And every headline gain flows through GPT-5.6-Sol as Builder; how much survives with a weaker Builder is exactly the question the transfer section raises without answering.