4M-Parameter Pixel Transformer Emerges as Physical World Model

Author @mathemagic1an conducted experiments to explore whether video models truly understand physics. He built a simple physics environment with collisions, wall bounces, and scoring, and trained a 4M-parameter pixel Transformer to predict next frames at a cost of $10. The research shows the model is not merely interpolating pixels or memorizing the training set, but forms internal representations of physical laws, which can be converted into an interactive world model.

Confirmed

• The model exhibits a distinct "Physics Emergence Zone" in middle layers where physical information becomes readable.
• Averaging layer-5 activations by motion direction and applying PCA reveals a clear ring structure, with adjacent directions in physical space remaining close in the model's internal representation.
• Activations can be linearly mapped to read X/Y coordinates, and deeper layers predict collisions several steps ahead.
• Holding out a specific collision scenario as unseen, the model still simulates results correctly, weakening the "memorization" explanation.
• Using Anthropic's Jacobian lens, the author extracted writable motion directions from a model that only saw pixels, and turned it into a keyboard-controllable mini-game.

Why it matters

• Revealing black-box mechanisms: Modern video models show strong object permanence and occlusion tracking. Understanding how physical information is represented internally helps better understand and control large video generation models.
• Potential for causal world models: The experiment shows that training a Transformer on raw pixels can yield a world model with causal logic, even playable like a game, offering new ideas for practical applications and controllable generation.

2026-07-29 ~ 2026-07-29 · 10 related posts

• [source] Dissecting Video Models with Jacobian Lens: 4M Params Emerge Causal World Model — mathemagic1an · 2026-07-29
• Anthropic’s J-lens turns a raw-pixel video model into a playable world model — mathemagic1an · 2026-07-29
• Modern video models track object permanence and occlusion surprisingly well — mathemagic1an · 2026-07-29
• [source] A 4M-parameter pixel transformer learns a tiny physics game for about $10 — mathemagic1an · 2026-07-29
• The real question is whether video models memorize or learn physical law — mathemagic1an · 2026-07-29
• Held-out collision scenes still generalize in the video model — mathemagic1an · 2026-07-29
• Video-model activations linearly expose position and predict collisions early — mathemagic1an · 2026-07-29
• [source] Anthropic Jacobian lens pulls a writeable motion direction out of a pixel-only model — mathemagic1an · 2026-07-29
• Motion direction forms a ring in a video model’s activation space — mathemagic1an · 2026-07-29
• Video models reveal a “Physics Emergence Zone” where motion direction becomes readable — mathemagic1an · 2026-07-29