ai& open-sources TermGrade: 1,004 terminal RL envs that lift Terminal-Bench by 3.1 points

DavidBennett__ · x · 2026-10-09

Japanese AI startup ai& released TermGrade, an open RL resource for terminal agents: 1,004 executable Linux environments (container + instructions + pytest verification) evaluated across 6 model configurations — Kimi-K3, DeepSeek-V4-Pro, GLM-5.2, Qwen3.6-27B, and gemma-4-31B-it with and without reasoning — covering 36,144 recorded runs (4.5B tokens, 20k compute hours), including 14,000 failures with per-test failure details.

Key findings and method:

Tasks are labeled with measured per-model success rates instead of human difficulty ratings; the '≈50% success' task sets differ drastically across models, so RL task selection must be measured with the target model, not borrowed from other models' results.
Models learn best from tasks they solve about half the time, since those yield both positive and contrastive negative signals.
RL on tasks selected this way for gemma-4-31B-it improved Terminal-Bench 2.1 by 3.1 points; 5 repeated runs averaged +2.1 points, all beating the base model.

The release includes the task set, all run logs, the training task list, and checkpoints, with an evaluation/selection pipeline designed for cheap re-runs as models iterate.

Original post →

ai& open-sources TermGrade: 1,004 terminal RL envs that lift Terminal-Bench by 3.1 points

More from coding & agent