Dissecting Video Models with Jacobian Lens: 4M Params Emerge Causal World Model
mathemagic1an · x · 2026-07-29
The author explores whether modern video models truly understand physics or are merely stochastic pixel parrots.
Using a simple physics simulation (collisions, wall bounces, scoring), the author trained a 4M parameter pixel transformer to predict the next frame. Surprisingly, with just $10 of compute, the model demonstrated exceptional predictive capabilities.
- Emergent Physical Representation: The model isn't just memorizing the training set; it forms a higher-level representation capturing entities and their relationships, correctly deducing physical outcomes in unseen scenarios.
- Applying the Jacobian Lens: Using Anthropic's J-lens, researchers could linearly decode precise X/Y coordinates directly from the model's activations. Deeper layers even showed the model "anticipating" collisions several steps in advance.
- Playable Causal World Model: The model naturally contains causal directions for motion. By writing to specific directions in the activation space and mapping them to keyboard keys, the video model can literally be played like an interactive video game.
Related event: 4M-Parameter Pixel Transformer Emerges as Physical World Model(10 posts)→
More from Fun
- Tesla Roadster Thruster Teardown: SpaceX COPV Tank Plus Optimus Actuator — XFreeze · 2026-10-16
- Pearl Chat Puts Gemini, ChatGPT, Claude and Kimi in One Video Call — weskambale · 2026-10-16
- Critic Calls Figure's Terminator-Style Robot Demo Publicity Over Substance — FuSheng_0306 · 2026-10-16
- AI-Animated Short: Trapped in a Crypt with a Crypto Bro — PolicemansBeard · 2026-10-16
- Gemini 4 Argon builds pure-JS Tesla Roadster animation in 4 hours — zacharynado · 2026-10-16
- Fake popup-ad images keep fooling the elderly: MIL bricked her new phone in a week — justalexoki · 2026-10-16