Jacobian Lens explained: a single Jacobian matrix already works as an interpretability lens
CatAstro_Piyush · x · 2026-11-02
Sida Liu attempts to explain Anthropic's Jacobian Lens (J-Lens), an interpretability tool that reads intermediate activations as vocabulary scores, revealing concepts involved in intermediate computation — including information the model doesn't immediately output.
- Mechanics: compute the Jacobian ∂h{L,t'}/∂h{ℓ,t} between a layer of interest and the final layer, aggregate across token positions and dataset examples into per-layer matrices Jℓ, then decode activations via softmax(WU·norm(Jℓ·h)).
- The existing mathematical theory only explains what survives averaging: strong, consistently oriented responses persist; disagreeing ones cancel.
- The author's small experiment on Qwen3.5-2B shows a single same-position Jacobian, with no averaging, already decodes activations from unrelated contexts — suggesting individual Jacobians carry reusable transport structure, a more basic question than averaging theory answers.
More from Research
- QwenBend: Qwen writes and runs Bend programs inside its own forward pass using 53M hand-set weights — sull · 2026-11-02
- Robotics needs its own CERN or Hubble to cut through the noise — DJiafei · 2026-11-02
- Hidden gems tool surfaces underrated AI papers like TinyStories and 1-bit LLMs — omarsar0 · 2026-11-02
- VRCFluid Simulation: GPU-powered water physics for VRChat creators — Michael_Moroz_ · 2026-11-02
- Deep RL book excerpt: the policy loss is fake — a surrogate with no physical meaning — burkov · 2026-11-02
- NBER Paper Proposes Coasian Stakeholder Theory: Defining Stakeholders by Shutdown Power — sebkrier · 2026-11-02