JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution
Guibin Zhang, Leo Lu, Fangzhou Xie, Kang Zhu, Junhao Wang, Zhifei Xie, Zhaochen Yu, Zihang Liu, Zhongxiang Sun, Qiankun Li, Yue Liao, Heng Chang, Xiaobin Hu, Qibing Ren, Wangchunshu Zhou, Shuicheng Yan
cs.CL, cs.LG
2026-08-26
JIT-Agent synthesizes a task-specific harness on the fly. Flash then scores 85.1 on DeepSearchQA vs GPT-5.6's 76.0; GLM-5.2 gains up to 20.2 points at 36% lower cost.
Agent quality is a model-harness pair. Memory compression, planning, the action loop, and which tools or skills are exposed often move scores more than the backbone weights. Most harness work is ahead-of-time: polish one durable scaffold on a stream of traces and hope it fits the next task. Wide search wants parallel evidence gathering; terminals want a thin ReAct loop; deep research wants working memory over retrieved sources; repo edits want the filesystem as state. The right harness is instance-specific. Precompiling one universal scaffold searches a large design space, burns trajectories, and still misses the next problem's structure.
JIT-Agent writes the harness as four composable modules, Memory, Planning, Action, and Capability orchestration, under a fixed protocol, so generation emits executable modules rather than unconstrained agent programs. HarnessFactory re-implements 13 existing scaffolds behind that interface as a seed bank.
Training has three stages. Stage I has a stronger teacher synthesize task-adapted harnesses; the student imitates valid ones, then learns preferences that raise reward without worsening latency or cost. Stage II turns compiler errors, interface mismatches, and runtime failures into repair traces of at most two rounds. Stage III runs Evo-GDPO: a group of candidates is scored against the archive incumbent, with reward, latency, and cost normalized on separate channels, and only frontier-improving designs enter the bank. At deployment the generator is frozen. It can emit several harnesses in parallel for one task, or retrieve and update the archive across a stream.
The generator is JIT-Agent-27B, trained from Qwen3.6-27B. Any off-the-shelf agentic LLM can sit underneath as the executor.
Swapping the harness with the backbone held fixed improves all 18 matched backbone-benchmark pairs. GLM-5.2 rises from 74.1 to 81.8 average (+7.7); DeepSeek-V4-Flash from 66.7 to 75.5 (+8.8). Flash reaches 85.1 on DeepSearchQA against GPT-5.6's 76.0, and 73.0 on OdysseyBench against 68.7. GLM-5.2 jumps 20.2 points on DeepPlanning-Travel (62.8 to 83.0); Flash jumps from 59.1 to 83.9 on shopping planning.
| Backbone | DeepSearchQA | xBench-DS | AgentIF |
| GLM-5.2 vanilla | 89.2 | 76.0 | 63.0 |
| JIT-Agent + GLM-5.2 | 93.9 | 88.0 | 69.9 |
| DeepSeek-V4-Flash vanilla | 76.2 | 70.1 | 58.4 |
| JIT-Agent + Flash | 85.1 | 82.0 | 63.8 |
| GPT-5.6 | 76.0 | 81.0 | 68.0 |
Against Claude Code, Codex, OpenCode, Hermes, and NanoBot on a fixed backbone, Flash scores 85.1 on DeepSearchQA versus NanoBot's 80.4, while tokens fall from 924K to 400K and cost from $0.131 to $0.066 per case. JIT is cheapest on tokens and dollars in all six controlled settings, cutting cost 14.9% to 54.1% versus the cheapest fixed harness (mean 36.0%). Across 24 family-variant comparisons versus ReAct the mean gain is 7.6. Streaming archive updates finish above independent static generation on shopping planning, travel planning, and OfficeBench.
Harness engineering becomes a trainable, transferable, test-time compounding skill rather than a hand-built durable artifact. For already strong open backbones, a generated scaffold can beat larger closed models on several agent tasks, with shorter and cheaper traces. Production runtimes need not rebuild the whole stack per query; generating memory, planning, and tool exposure per task is already enough to move the cost-performance frontier.
The paper does not table an ablation that drops individual training stages, so each stage's contribution is inferred from the recipe. Streaming evolution is shown as cumulative-accuracy curves without a numeric endpoint table. Generated harnesses still pass a protocol validator and bounded repair; stability on messy runtimes depends on the executor actually following the protocol. The authors note that deployed systems will more likely keep a stable core and swap selected components than redesign the entire scaffold every time.