Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?
Yibo Li, Jinhang Qiu, Zhi Zheng, Qianyun Guo, Jiaying Wu, Shuo Ji, Bryan Hooi
cs.AI
2026-10-06
NUS tests agents on 20 counterintuitive text games. On Kimi K3, keeping raw logs yields learning gain +16.9, ahead of methods that compress experience into rules.
A higher score next time does not show that an agent learned a new rule. Benchmarks for self-evolving agents mostly use tasks whose rules are already in the instructions, or settings pretraining has likely seen: Jericho games, successive software issues, web navigation. When the score rises, the gain can be knowledge the model already had.
Learn2Play Bench, from the National University of Singapore, is 20 hand-written text games with rules that are new or counterintuitive. In SynergyCorp, stealing a colleague's lunch helps a promotion and working hard does not. The prompt never says so. The model acts, reads text feedback, and carries that record into later episodes. Weights stay frozen. Outcomes are deterministic and scored automatically.
Ten fixed games repeat one instance. Ten re-shuffled games change characters, menus, or layout and keep the hidden rule. Five challenge games, Catnip, DreamGarden, Mola Tea, RedEye, and Roadside Observatory, run 10 episodes at weight 3. The other 15 run 5 episodes at weight 1. Each setup plays three trials that do not share memory. Raw scores are divided by a reference score and scaled to 100. Max averages the peak of each trial, so peaks of 40, 60, and 80 become 60. Learning gain (LG) is the gap between late and early mean scores. Learning slope (LS) is the per-episode trend of the mean score.
Six memory styles are compared with the backbone held fixed. Naive keeps nothing across episodes. Memory appends the raw transcript and drops the oldest text only when the context is full. Reflexion and ReasoningBank compress a trajectory into lessons. EvoTest rewrites a guiding prompt and a state-extraction program. AWM builds workflows from successful episodes only. Eleven backbones share OpenCode. Opus 5 is also run in Claude Code, and GPT-5.6-SOL in Codex. Human players are college students, 21 to 30 per game, with no AI used during play.
| Setup | Metric | Score |
| Opus 5.5 / OpenCode | Max / Mean | 80.1 / 61.0 |
| Qwen 3.8 Max / OpenCode | LG / LS | +25.2 / +6.30 |
| Kimi K3 + Memory | Max / LG | 63.3 / +16.9 |
| Kimi K3 + EvoTest | Max / LG | 62.4 / +1.8 |
| Human Top-1 | Max / Mean | 84.3 / 53.5 |
| Opus 5, Claude Code vs OpenCode | Max | 81.8 vs 74.5 |
Under OpenCode, Opus 5.5 has the highest Max and Mean. Qwen 3.8 Max improves fastest. GPT-5 Mini's Max is 34.0. On Kimi K3, Memory leads all four metrics. Naive sits at Max 54.7 and LG +0.3. EvoTest's Max is close, but its LG stays at +1.8. In one EvoTest run with Opus 5 on GemForge, a single failed fusion is written into the prompt as a ban on many combinations never tried. Averaged across Kimi K3, Opus 5, and GPT-5 Mini, Memory still has the best LG, +11.4, while Max goes to Reflexion at 60.0. On Opus 5 alone the order flips: EvoTest reaches Max 66.1, Memory 60.9. OpenCode with that same Opus 5 still scores 74.5. One reply can reread feedback, run a calculation, and then act. The self-evolving loops call the model once per action.
Human Top-1 Max is 84.3, above every agent configuration here, but Mean is 53.5, below Claude Code's 62.3. Across all human players, Max is 43.6. On fixed games, consecutive action sequences are 0.54 similar for humans and 0.66 for agents. After a worse episode, humans set a new personal best next time in 13 of 39 cases, agents in 52 of 238. When the surface changes and the rule does not, Sonnet 5's LS falls from 4.90 to 2.23. Opus 5 falls from 4.10 to 3.51. Kimi K3 has a higher token price than Sonnet 5, yet OpenCode uses about 0.56M tokens and $0.313 per episode for Max 68.5, against Sonnet 5 at 0.73M tokens, $1.33, and Max 55.9. Claude Code with Opus 5 costs about $0.647 per episode. OpenCode with Opus 5 costs about $2.57.
The best score and the fastest learning are different models. A summary written from thin evidence can freeze a bad rule, and later episodes keep following it. The raw transcript is still there when the next observation disagrees. In Haunted Inn, Opus 5 can state the room taboos and still assign safe rooms greedily, leaving a later guest with nowhere to stay.
The harness moves the result more than the choice of summarizer. Claude Code lifts Opus 5 from Max 74.5 to 81.8 and LG from +21.3 to +27.1, at a lower estimated bill. Codex lifts GPT-5.6-SOL from Max 57.0 to 74.3. These models are trained for their own tool loop. Extra lessons do not repair a mismatched shell.
The human lead is Top-1 selected by peak score. The mean over all participants is Max 43.6, below every OpenCode backbone except GPT-5 Mini. Agent Max averages three trial peaks. Human Top-1 is one person's best record. The similarity gap, 0.54 versus 0.66, and the recovery counts, 13/39 versus 52/238, cover fixed games only. Repeated episodes are not independent, and the appendix treats both figures as descriptive.
Memory wins all four metrics on Kimi K3. On Opus 5, EvoTest scores higher, and across three backbones Reflexion has the higher Max. The appendix says the GemForge ban does not prove that compression caused the gap versus Memory. Reference scores for Catnip, DreamGarden, Mola Tea, and Patch Reality are estimates. Challenge games were chosen in a pilot and then weighted three times as heavily. The main table uses one seed per template. GPT-6 Astra is a confirmed $150 bill over 600 episodes. Other costs are list-price estimates, and some token logs are missing. The study never updates weights.