S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?
Jiajun Shi, Siyuan Tao, Yuhao Wu, Zexuan Wang, Jingyuan Zhang, Jiaheng Liu, Xinping Lei, Xinrong Zhang, Siyuan Fang, Zhewen Tan, Tianle Cai, Junhao Fang, Jiameng Huang, Yueyang Wang, Jinkai Liu, Yuxuan Zhang, Jian Yang, Zhoujun Li, Shen Yan, Wenhao Huang, Ge Zhang
cs.CL
2026-09-01
S3Gym tests judging and improvement on seven games. Summaries help when rules compress; history wins on local boards. Qwen3-8B SFT lifts Trust 0 to 30 and drops PvZ 23 to 6.
Most agent benchmarks treat the model as a frozen policy: give it a task, give it an environment, score this attempt. Deployment asks a different question. Can the agent test itself, judge the traces, and turn that experience into better later decisions? Collecting trajectories is not learning. Experience has to be examined, abstracted, and reused.
S3Gym splits that loop into Self-Testing, Self-Judging, and Self-Improvement, and hides the verifier score from the agent. Exploration uses looser rules; evaluation is stricter, with disjoint seeds. The agent must organize experience with its own scores. The benchmark then measures transfer with an executable verifier.
Seven text games with programmatic checkers cover different structures: Chess, Minesweeper, Nullify, Tetris, Snake, Plants-vs-Zombies, and Trust Evolution. Exploration is permissive, evaluation is harsher. Minesweeper exploration has two extra lives; evaluation ends on the first mine. Snake treats collisions as no-ops in exploration and as terminal in evaluation.
At each step the model emits an action and a self-assigned immediate score. The environment returns observable feedback; verifier rewards stay on the benchmark side. Experience is reused in three ways:
The main suite evaluates GPT-4o, GPT-4.1, o3-mini, Gemini-2.5-Flash/Pro, GPT-5.5, and Gemini-3.5-Flash. Agents play 30 exploration episodes and, every 3, run 3 strict evaluation episodes. Metrics are mean score, max, and AUC+ above the initial score. A separate training path fine-tunes Qwen3-8B across 20 checkpoints and evaluates without history or summaries.
Context-level gains are neither automatic nor uniform. Summaries help when experience compresses into rules: Tetris, Nullify, and Trust prefer Summary on mean NABA. Minesweeper, PvZ, and Snake, which depend on local board state, often prefer raw history. GPT-5.5’s PvZ AUC+ falls from 548.499 under History to 33.219 under Summary. Gemini-3.5-Flash’s Chess AUC+ falls from 12.280 to 0.
Self-Judging is only half reliable. Across 98 runs and 116,117 transitions, Chess event agreement is 0.496 with over-confidence 0.365. PvZ agreement is 0.881, yet normalized MAE is 0.882, so magnitudes are uncalibrated. Judgment quality barely predicts the next checkpoint’s score change: ρ(A, g) = −0.010.
Training Qwen3-8B lifts Trust Evolution from 0 to a max of 30, staying above baseline at 18 of 19 updates, AUC+ 163.5. Minesweeper, Nullify, and Tetris stay at 0. PvZ drops from 23 to 6 at every updated checkpoint. Snake touches 1 at a few epochs and returns to 0.
For anyone building self-improving agents, the split between “can score” and “can improve” is the useful part. Local judgments can be right while the next evaluation does not move. Summaries pay off when a stable rule exists, and erase the state-contingent details that reactive games need. Parameter updates can store a strategy, and can also write a bad exploration policy into the weights.
Unlike Reflexion or Voyager, which exhibit one mechanism, S3Gym compares three pathways under one environment and one budget, then tests on a stricter hold-out. The diagnosis is that converting feedback into an executable, transferable policy is the missing step.
Score scales differ by orders of magnitude; AUC+ cannot be summed across games, and the paper says so. The main table only includes proprietary models with all seven games complete. Training is a single Qwen3-8B line, so instability may be model-specific. The exploration–evaluation gap is a designed difficulty jump; how much negative transfer comes from that jump versus bad judging is not separated.
The judgment–improvement coupling is correlational. Chess is 5 steps on a 15×15 board, then 22 pieces at eval, closer to short-horizon state prediction than to actual chess. There is no human or specialist RL baseline, so absolute scores are hard to read.