World Modeling in Transformers
Pierre Beckmann, Matthieu Queloz, Andre Freitas
cs.AI, cs.CL
2026-09-18
TaxiGPT encodes a faithful Manhattan map and a goal compass; 8.5% illegal turns under stress come from superposed intersection features, not a spaghetti map.
Vafa et al. (2024) trained GPT-2-style transformers on taxi rides through Manhattan, a setup the new paper calls TaxiGPT. The task is next-turn prediction given origin, destination, and the turns so far. A model trained on random walks takes a legal turn 99% of the time. Reconstructing a map from its outputs produced spaghetti, and forcing the taxi away from the goal three-quarters of the time wrecked route quality. The original conclusion was that the transformer was far from recovering New York's street grid.
Behavior underdetermines mechanism. Spaghetti maps can mean a garbled internal layout, or a faithful layout with a localization system that slips. This paper opens TaxiGPT and tests the second story.
The main model is GPT-2 XL (48 layers, 1600-d) trained on random walks. Intersection features are difference-in-means vectors: the mean residual when a node is current, minus the global mean. They decode better than linear probes. A minimal teleportation edit subtracts the current-node feature and adds a neighbor of the goal, then checks whether greedy decoding takes the legal step from the new node into the goal.
Street connectivity is tested two ways. A steering test injects an intersection feature, feeds a legal move, and asks which intersection feature grows most. A probing test trains a linear probe per move to predict the successor's feature. For navigation, the authors isolate a circular goal compass in the residual stream that encodes bearing to the destination. Ablating it keeps legal turns but wrecks long-range goal reaching.
The 4,516 intersection directions occupy a small subspace and cluster by legal-move set: intersections that permit the same next turns sit nearby. The paper calls this affordance packing. Localization reads a look-back window of recent position features rather than recomputing location from the full turn sequence.
The internal map is highly faithful.
| Metric | Result | Contrast |
| Intersections decoded at ≥90% accuracy | 99.6% (layer 18) | 12.7% for shortest-path training |
| Teleport succeeds on ≥1 scene | 99.3% of testable nodes | 8.0% with random node features |
| Street steering top-1 / top-5 | 76.6% / 93.3% | no external baseline |
| Successor probe | 89.2% | no external baseline |
| Legal-move set recovered from features | 99.4% | no external baseline |
| Goal-compass median angular error | 18.1° | clamping north/south makes the taxi follow |
Ninety percent of intersection variance sits in 244 dimensions; mean nearest-neighbor angle is 48.1°. For 91% of intersections, the feature is closer to its own legal-move group mean than to any other group.
The stress test samples origin–destination pairs a median 32 moves apart, while training rides start 99 moves from the goal, so the test is out of distribution. The model still reaches 81% of goals and current-node decode stays at 99%, but 8.5% of rides take an off-graph turn. Of those illegal turns, 93.5% fall into four modes: fatal slip 25%, silent leak 24%, full corruption 29%, give-up slip 16%. The shared cause is a weaker true-position write plus noise in the superposed subspace, so a wrong node becomes most active. Reinforcing the correct position at layer 11 lifts stress legality from 91.5% to 97.3%, detour success from 63.1% to 71.9%, and compression from 0.524 to 0.691. The map remains usable; recruitment is the failure.
Random-walk data matter more than architecture. Shortest-path next-token models decode only 12.7% of intersections. The same random-walk data at 384-d still decode 99.7%. A NextLat objective (predict the next latent) encodes streets at 94.8%, versus 81.9% for ordinary next-token. During training, legal-move and compass features mature before precise localization. Smaller models score better on behavior; the large model scores better on the causal indicator. The small random-walk model slips about 3× less, NextLat about 6× less.
Asking whether a model "has a world model" mixes three capacities: representing environmental structure, locating the current state, and using those representations to choose actions. They need not arrive together. Mechanistic work on Othello, mazes, and chess already found internal boards. This paper adds a sharper warning: a faithful map plus localization failure looks, from the outside, like no map at all.
For interpretability, the decode indicator is finer than behavioral scores because it tests both whether intersections are distinguished and whether the right one is recruited at inference. For training world models, data beat architecture in this sandbox. Random walks beat shortest paths; a next-latent objective pushes the model to store street transitions.
This is a case study, not a new architecture.
The authors note that TaxiGPT lives in a finite-state world with deterministic transitions; which extra capacities richer environments need is left open. Stress, detour, and compression tests are all reread as the same localization failure, but the compression rebuttal inspects only 146 prefix pairs.
The shortest-path rows in the comparison table may confound data with model size and training budget, so they should not be read as "data is everything." Affordance packing is presented as a protective mechanism; it may also be a side effect of the legal-move supervision. Manhattan's grid and the random-walk statistics are special. Extrapolation to world modeling in natural language should stay cautious.