TALES: Text Adventure Learning Environment Suite
Christopher Zhang Cui, Xingdi Yuan, Ziang Xiao, Prithviraj Ammanabrolu, Marc-Alexandre Côté
cs.AI, cs.CL
2025-04-19
Microsoft and UCSD unify five text-adventure frameworks into TALES, an LLM reasoning benchmark. 34 models play zero-shot: near-perfect on synthetic games, under 15% on human-written ones.
Single-question benchmarks can no longer tell you whether a model can actually reason. What tests a model is a chain of dependent decisions: go to the kitchen, grab the pot, light the burner, add the ingredient, each step built on the observation from the last and course-correcting when something goes wrong. Text-adventure games are a natural testbed for this, pure text in and out, hidden state, long horizons, and for decades they have been a classic arena for agents.
The problem is that the arena's exam papers are scattered. TextWorld, ALFWorld, ScienceWorld, and Jericho each come with their own setup, and each hands the agent a different amount of scaffolding: action templates here, injected expert knowledge there. Pile on enough scaffolding and the score rises, and you can no longer tell whether the model reasons or just follows hints. TALES folds these papers into one protocol built on minimal scaffolding, so models sit the exam with no cheat sheet.
TALES pulls in five text-adventure frameworks as-is, 122 games in total: TextWorld (10 cooking games by difficulty), TextWorldExpress (16, a faster reimplementation), ALFWorld (12 household tasks), ScienceWorld (30 elementary-school science tasks), and Jericho (54 human-authored interactive-fiction games such as Zork). The first four are procedurally generated; Jericho is written by people for people to play.
The centerpiece is a prerequisite called Simon Says. The basic version asks the agent to echo a command; Simon Says With Memory hands it the full walkthrough up front, awards a point per correct action, and resets the whole game on any wrong or out-of-order step. It sounds trivial, but it tests one basic thing: can the model follow instructions faithfully over a long horizon? Simon Says scores correlate with overall TALES performance at Pearson r = 0.83, so the authors set a gate, and only models scoring at least 90% on the 100-step version enter the main evaluation. Nine of 34 cleared it.
The agent itself is zero-shot, with a minimal system prompt: you are playing a text game, return one short action per turn, and type "help" when stuck. No environment-specific expert knowledge. The bare setting is deliberate; strip away templates and hints and the score reflects the model's own reasoning. Each game is capped at 100 steps, a compute- and money-driven limit that leaves only 57% of Jericho's maximum score reachable.
Of the models that cleared the gate, the reasoning model Claude-3.7-Sonnet leads overall at 52.5, with Claude-3.5-Sonnet the strongest zero-shot entry at 50.4, then gemini-2.5-pro-preview (49.3), o1 (44.2), and gpt-4o (40.6).
There is a clear cliff between synthetic and human-written games. The same models that score 97.3 on TextWorld drop to 12.5 on Jericho; gpt-4o falls from 83.6 to 5.6.
| Model | TextWorld | ALFWorld | ScienceWorld | Jericho |
| Claude-3.7-Sonnet | 97.3 | 83.3 | 76.5 | 12.5 |
| gpt-4o | 83.6 | 56.7 | 61.4 | 5.6 |
Jericho, the human-written games built for entertainment, is where models collectively break down: the best agent averages under 15% on this category. Even on Zork1, the most famous of them, where the authors see Zork references repeatedly in Claude-3.7's chain of thought and strongly suspect it leaked into training data, no model reaches half of the score reachable within 100 steps. The paper's line is blunt: parametric knowledge does not make up for a lack of reasoning.
The cost is steep too. Reasoning models burn tens of millions of tokens on a single Jericho game, with DeepSeek-R1 using the most.
For anyone building agents, TALES is a relatively honest mirror. Scoring near-perfect on a scaffolded synthetic benchmark does not mean a model can reason independently through a long-horizon, lightly prompted, human-designed task. The paper nails this with data: the same models fall off a cliff from synthetic games to Jericho, and the break lands on grounded and inductive reasoning.
The Simon Says gate is also practically useful. It is cheap and reproducible, and with a 0.83 correlation to overall scores it works as a quick pre-filter. A model that cannot even follow a given walkthrough is not worth spending the budget to run all 122 games.
The 100-step cap is a real constraint. Jericho can yield at most 57% of its max under it, so "under 15%" is partly pressed down by the step ceiling; the authors concede compute and budget forced it.
Minimal scaffolding is a design choice, not a neutral one. Other benchmarks give agents action templates and expert knowledge for good reasons, and TALES strips them all out, which is rough on methods built around scaffolding. Treat cross-comparisons with that in mind.
A couple of models ran a single seed (gemini-2.5-pro-preview, gemini-2.0-flash), so variance is uncontrolled. Data contamination cannot be ruled out either: classics like Zork likely sit in the training set, and the authors can only "strongly suspect" it from chain-of-thought references, not quantify it.
The sharpest caveat: the authors deeply analyze the traces of only one reasoning model, Claude-3.7-Sonnet, so the conclusions about grounded and inductive reasoning failing first, and traces contradicting across steps, are read off a single model. They also discard the thinking trace each step rather than using extended thinking, so some of that "traces contradict" finding may be an artifact of the setup, not the model.