Interpretability researchers ask why models rarely learn exact, readable algorithms
burny_tech · x · 2026-10-27
burnytech argues that interpretability work should surface more exact, human-readable mechanisms — like the trigonometric algorithm for grokking modular addition or line-breaking detectors in toy model papers — reacting to the paper "When Models Manipulate Manifolds: The Geometry of a Counting Task." That paper shows LLMs perceive visual text properties (line width, char counts) from raw number sequences: attention builds a count manifold, QK twists it to sense end-of-line, and a linear boundary predicts overflow. His question: where are the crisp algorithmic solutions?
Related event: Transformer Circuits Paper Examines Manifold Geometry of Counting(3 posts)→
More from Research
- Kevin Bryan: maximizing raw accuracy is rarely the right way to train AI assistants — Afinetheorem · 2026-10-27
- Szegedy: a flaw in common training losses went unnoticed for over a decade — ChrSzegedy · 2026-10-27
- Claude research model claims to refute Exponential Time Hypothesis with 14M-token autonomous run — scaling01 · 2026-10-27
- "ETH is false": CS community buzzing over claim that upends SAT exponential-time assumption — akbirthko · 2026-10-27
- RL environments essay: knowing how to operate software isn't knowing how to achieve goals — ShenRaphael · 2026-10-27
- Δτ-Bench: Agents Face a Moving Tool Frontier, Best Model Scores Just 42.3% — UniversityCollegeLondon · 2026-10-27