End-to-End Sparse Dictionary Learning Makes SAE Features Functionally Important
burny_tech · x · 2026-10-20
This post shares the arXiv paper on end-to-end (e2e) sparse dictionary learning. Standard sparse autoencoders reconstruct internal activations but may capture dataset structure rather than the network's computational structure. E2E SAEs train by minimizing the KL divergence between the original model's outputs and those of the model with SAE activations inserted, ensuring learned features are functionally important. The result is a Pareto improvement over standard SAEs: more network performance explained, fewer total features, fewer active features per datapoint, with no interpretability cost. Training library and analysis code are open-sourced.
More from Research
- Why GRPO collapses on multi-turn agent training, and how to stabilize advantage estimation — burkov · 2026-10-20
- SJTU's MSAR Residual Architecture Delivers 1.58x Iso-Loss Training Speedup Over Standard Residuals — burny_tech · 2026-10-20
- From memory capacity to transformers: outer-product stores cap out near 2×key dimension — burny_tech · 2026-10-20
- UIUC paper: Fisher-selected 0.1% of parameters recovers 85% of full RL training gains — burny_tech · 2026-10-20
- Frontier models already saturate persimmon, a tiny human-distribution benchmark — noahdgoodman · 2026-10-20
- New AI model claims to be the first to employ visual thinking — LinkedInNews · 2026-10-20