Paper: Transformers Do Have Faithful World Models, Just Scrambled

A new arXiv paper trains taxiGPT on Manhattan random-walk data and finds via interpretability analysis that the model internally encodes a faithful world map, suggesting Transformers do learn world models but feature superposition interferes, causing faithful internal representations alongside flawed behavior.

2026-09-23 ~ 2026-09-23 · 3 related posts

• Interpretability paper finds a faithful internal world model inside taxiGPT — __nmca__ · 2026-09-23
• Paper: Transformers do have world models — failures trace to feature interference — burny_tech · 2026-09-23
• New Paper: Transformers Hold Faithful Internal World Maps but Fail to Act on Them — xuanalogue · 2026-09-23