AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses
Cheng Qian, Wenting Zhao, Liangwei Yang, Heng Wang, Jielin Qiu, Heng Ji, Silvio Savarese, Huan Wang, Shelby Heinecke
cs.LG, cs.AI, cs.CL
2026-08-13
A stronger model builds inference-time harnesses for a weaker one with no parameter updates, nearly doubling Theory-of-Mind accuracy from 0.49 to 0.91 by offloading unstable reasoning into deterministic code.
Distillation usually moves capability from a large teacher into a smaller model by updating the smaller model's weights, through teacher forcing or on-policy distillation. That works but is expensive, and once training is done the model is frozen. This paper asks a different question: can a strong model transfer its capability to a weaker one at test time, without touching the weaker model's parameters?
The setup is that a strong builder model (say GPT-5.5) writes an inference-time harness, a layer of program logic wrapped around the target model, then hands it to a weaker target (say GPT-5.4-mini) to answer with. The builder uses only 5% of the data as a validation set and refines the harness over several rounds before evaluation on the full test set.
The test bed is four Theory-of-Mind benchmarks (BigToM, Hi-ToM, MMToM-QA, MuMA-ToM) that probe whether a model can infer agents' beliefs, goals, and actions.
The builder receives the task rules, a demo of how to call the target model, and the 5% validation set. Its loop: propose or revise harness code, run the target model on the validation set, diagnose where it fails, and fix it. When done it exports a test-time entry point.
A harness is a mix of prompt templates and few-shot exemplars, routing logic that dispatches items by subtype, deterministic solvers that hand symbolic reasoning to code, and format enforcement with answer verification.
The important finding is not how complex the harness gets but where the gains come from. They come from offloading unstable reasoning into deterministic code, subtype routing per benchmark, and strict answer-format enforcement. Letting the target reason longer or sample more contributes little. The harness works by boxing the model in, not by making it smarter.
The target model GPT-5.4-mini scores 0.488 raw. The mean over all scaffolded runs is 0.763 (+0.275), the best run reaches 0.912 (+0.423, an 86.7% relative gain), close to a hand-designed harness at 0.939.
Per benchmark (best scaffold):
| Benchmark | Scaffolded | Raw |
| BigToM | 1.00 | 0.503 |
| Hi-ToM | 0.803 | 0.569 |
| MMToM-QA | 0.842 | 0.412 |
| MuMA-ToM | 0.857 | 0.469 |
Three regularities stand out. Weaker targets gain the most: GPT-5.4-mini rises 0.262 on average, while the already-stronger Gemini-3.5-flash rises only 0.110 and actually regresses on Hi-ToM and MuMA-ToM. Builder reasoning effort helps monotonically (Spearman ρ=0.77; low/medium/high/extra-high effort scores 0.711/0.793/0.807/0.856). Platform effects are small: a native platform edge averages just +0.013 and is not significant, and the builder's own capability dominates.
For practitioners this is a complement to retraining. If you run a closed weak model whose weights you cannot change, or you do not want to retrain after launch, a strong model can build a task-specific harness offline and you hang it on the target. Low cost, reusable, zero training.
Its task boundary is also clear. It fits tasks with reusable structure, like the belief reasoning in ToM benchmarks that can route to deterministic sub-logic, and does not fit nested higher-order belief tracking under deception. It is closer to freezing procedural work than to making the model smarter.
The authors acknowledge that the strongest harness still struggles with higher-order Hi-ToM cases (recursion depth ≥2; accuracy drops to 0.70 at order 4), deception scenarios, Bayesian goal-inference subtypes in MMToM-QA, and social-goal labels in MuMA-ToM.
The bigger worry is over-scaffolding: for an already-strong target, a harness can perturb answers the model would otherwise get right. The method leans on benchmark-specific rules, and a single logic error can shift accuracy by tens of points, so robustness is open to doubt. Across runs the mean standard deviation is 0.036, fairly stable but not fully deterministic.