Anthropic Jacobian lens pulls a writeable motion direction out of a pixel-only model

mathemagic1an · x · 2026-07-29

Using Anthropic’s Jacobian lens, the author shows how to recover a causal motion direction from a pixel-only transformer and turn it into a playable control signal.

The model never saw controllers or actions—only pixels.
By tracing how activations affect downstream frames, the author extracts reusable motion directions from the residual stream.
Those directions can be written back into the model to steer ball velocity and mapped to keyboard input.
The post frames this as evidence that motion is represented causally inside the network, not merely as a readable but non-writeable feature.

Original post →

Anthropic Jacobian lens pulls a writeable motion direction out of a pixel-only model

More from Multimodal