CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks
Fanzhe Meng, Guoxin Chen, Jiale Zhao, Shuang Sun, Zhiyu Lin, Wayne Xin Zhao, Ruihua Song, Ji-Rong Wen, Kai Jia
cs.LG, cs.CL
2026-08-07
CalibForge turns solver pass/fail outcomes into construction feedback, revising terminal tasks into a learnable zone; models trained on 5,431 calibrated tasks reach 47.57% on Terminal-Bench 2.0.
Training terminal agents (coding agents that work in a shell) needs lots of executable, verifiable tasks. Today's synthesis pipelines can build tasks and pass them through validation, but "it runs" does not mean "it teaches": tasks that are too easy give no gradient, and tasks that are unsolvable give none either. Whether a task lands in the learnable band is invisible to construction and validation alone; you only learn it by having solvers actually attempt it.
CalibForge's idea: use solver behavior as construction-time feedback, and push tasks toward the learnable band.
CalibForge turns task construction into a constrained author and solver adversarial loop. From a clue, an authoring agent does web research (docs, GitHub issues, Stack Overflow), picks a concrete engineering problem (version-specific bug, dependency conflict, config pitfall, reproducible edge case), and jointly writes the task instruction, a Docker execution environment, and verification tests. Each candidate passes two gates: structural validation (files present, environment builds, all tests fail in the initial state) and self-solving (the author can solve it in a sandbox), proving it is executable and solvable.
Then comes adversarial calibration. Each round spins up isolated sandboxes, dispatches solver subagents on the same task, judges pass or fail with the verifier, and feeds the outcomes, full trajectories, and diagnostic summaries back to the author. The author revises the task (it can redo research, change the instruction, environment, or tests), revalidates, and reprobes until a retention criterion is met or 50 rounds run out. Two criteria define the learnable zone:
Both are anchored in demonstrated solvability, so retained tasks are never unsolvable. A strong teacher (DeepSeek-V4-Pro) then distills SFT trajectories from retained tasks to fine-tune the target agent.
CalibForge produces 5,431 calibrated tasks (1,263 multi-solver, 4,168 contrastive) across 16 domain categories.
On Terminal-Bench 2.0, models trained on CalibForge trajectories reach 32.58% (Qwen3-30B-A3B-Instruct) and 47.57% (Qwen3.5-35B-A3B), beating the strongest baselines under the shared protocol by 6.36 and 6.75 points. Every baseline task set is re-distilled with the same teacher and recipe, so the difference comes from the data itself.
Transfer to out-of-distribution software-engineering benchmarks is large on the weaker backbone: on SWE-bench Pro the 30B model jumps 27.68 points (3.26 to 30.94), the 35B 3.03; on Doc2Repo the 30B jumps 30.04, the 35B 3.85. The 35B's relative gains are much smaller than the 30B's, so stronger bases see diminishing returns from calibrated data.
The ablation is the clearest evidence. At a matched 1,300 tasks: authoring and validation alone give 22.47%; adding single-solver feedback gives 24.34% (up 1.87); multi-solver calibration 29.21% (up 6.74); contrastive calibration 31.09% (up 8.62). Multi-solver produces fewer trajectories than no calibration yet scores far higher, so the gain is not about volume.
One detail: at the first contrastive probe, only 19% of tasks naturally satisfy the strong-pass and weak-fail relation; the rest passed structural validation and self-solving but fell outside the band (mostly too easy, both passing). After revision and reprobing, 96% satisfy it, showing that solver feedback reshapes tasks rather than merely filtering them.
Terminal and coding agents are among the hottest directions, and the scarcity of high-quality verifiable tasks is an acknowledged bottleneck. CalibForge defines task difficulty as a solver-relative learnable zone and uses an adversarial loop to drive tasks into it, giving a reusable recipe for building agent training data.
The paper has no dedicated limitations section. A few concerns stand out. The pipeline is expensive: the authoring agent is DeepSeek-V4-Pro, each task runs up to 50 calibration rounds with multiple solvers, and each attempt can take 100 steps and 30 minutes, so producing 5,431 tasks is not cheap. Relative gains shrink markedly on the stronger 35B base, and whether the learnable zone keeps yielding gains for already-strong models is unverified. All solvers and the teacher come from a few closed models, so the difficulty band is tied to their capability distribution; a different solver pool could shift the conclusions.