BeFOND: encoder-free sparse coding matches Gemma Scope with 1000x fewer training tokens
gottapatchemall · x · 2026-10-08
Researchers introduce BeFOND, an encoder-free iterative sparse coding model for interpreting LLM internals. Highlights: closed-form inference and learning rules unified as natural gradient flow on free energy; at 524k width it matches Gemma Scope's single-feature probing accuracy with over 1000x fewer samples (6M vs 8B SAE-training tokens); it keeps improving with SAE width while alternatives plateau; and the authors provide theory explaining why it works. Detailed 17-tweet thread included.
More from Research
- Harvard paper: higher SWE-Bench scores didn't translate into more merged pull requests — rohanpaul_ai · 2026-11-23
- Princeton's SSoN Benchmark: Best Vision Model Scores 39.7% vs Human 83.3% on Spatial Consistency — jiqizhixin · 2026-11-23
- Four Color Theorem Was Exhaustion — This Lean Proof Is Different, Argues Mathematician — gleech · 2026-11-23
- Samsung Labs open-sources LittleBit, compressing LLM weights to 0.1 bits — MikePFrank · 2026-11-23
- Mathematicians Debate: Can 60GB of Lean Code Turn Into Real Understanding? — gleech · 2026-11-23
- Adversarial examples aren't ubiquitous: budget matters, argues researcher — maksym_andr · 2026-11-23