E2E Sparse Dictionary Learning: SAE Features That Are Actually Functionally Important
burny_tech · x · 2026-10-20
The arXiv paper (2405.12241) by Dan Braun et al. proposes end-to-end sparse dictionary learning for SAEs. Instead of reconstructing activations, e2e SAEs minimize the KL divergence between the original model's output distribution and that of the model with SAE activations inserted, ensuring learned features are functionally important rather than merely reflecting dataset structure. Compared to standard SAEs, e2e SAEs explain more network performance with fewer total and fewer simultaneously active features, at no interpretability cost. The training library and analysis code are released openly.
More from Research
- Foundation model trained on 40M face photos predicts cancer outcomes and biological age — EricTopol · 2026-10-20
- Cloudflare introduces clairic: a category-free model paradigm for extracting values from raw inputs — ritakozlov · 2026-10-20
- Why GRPO collapses on multi-turn agent training, and how to stabilize advantage estimation — burkov · 2026-10-20
- SJTU's MSAR Residual Architecture Delivers 1.58x Iso-Loss Training Speedup Over Standard Residuals — burny_tech · 2026-10-20
- From memory capacity to transformers: outer-product stores cap out near 2×key dimension — burny_tech · 2026-10-20
- UIUC paper: Fisher-selected 0.1% of parameters recovers 85% of full RL training gains — burny_tech · 2026-10-20