Anthropic Jacobian lens pulls a writeable motion direction out of a pixel-only model
mathemagic1an · x · 2026-07-29
Using Anthropic’s Jacobian lens, the author shows how to recover a causal motion direction from a pixel-only transformer and turn it into a playable control signal.
- The model never saw controllers or actions—only pixels.
- By tracing how activations affect downstream frames, the author extracts reusable motion directions from the residual stream.
- Those directions can be written back into the model to steer ball velocity and mapped to keyboard input.
- The post frames this as evidence that motion is represented causally inside the network, not merely as a readable but non-writeable feature.
More from Multimodal
- AI video turns a kidnapping premise into a self-duplication joke — umesh_ai · 2026-07-29
- Sonder Editor brings an open-source video timeline to ComfyUI — SonderSaid · 2026-07-29
- Claude gaming art leaps: year-over-year comparison is stunning — iamfakhrealam · 2026-07-29
- SDXL Image Gen: How to Build Complex Multi-Character POV Interactions — ZeHirMan · 2026-07-29
- Hailuo AI’s new video model reportedly handles up to 12 image, video, and audio references — aziz4ai · 2026-07-29
- PDD accelerates image and video diffusion by predicting multiple denoising steps at once — ArashVahdat · 2026-07-29