Revisiting why transformers can't do zero-shot recursion, via Transformer Circuits math
akbirthko · x · 2026-10-31
- The author revisits their old thread on why transformers struggle with zero-shot recursion and can't maintain internal state like an RNN, admitting some points were wrong but the core held up.
- Key reading: Anthropic's 'A Mathematical Framework for Transformer Circuits' (Elhage, Nanda, Olah et al., 2021), which reverse-engineers transformer computations for mechanistic interpretability.
- A good entry path for understanding transformer circuits and the limits of GPT-style architectures.
More from Research
- Zip-NeRF Gets Orders of Magnitude Faster to Train, but Rendering Still Crawls at 3-4 FPS — jonstephens85 · 2026-10-31
- Researchers reconstruct Meta Muse's hidden charter from 884 chain-of-thought samples — sethlazar · 2026-10-31
- 14 AI models tested on 8,000 startups: none beats simply ranking by hiring growth — HeyAmit_ · 2026-10-31
- Working Paper Argues AI Can Break the Coasian Ceiling on Firm Size — zetalyrae · 2026-10-31
- Molence Bio Open-Sources Mol-1, Matching Closed SOTA at Antibody Structure Prediction — AllThingsApx · 2026-10-31
- AI-assisted proof: first high-probability regret bounds for Feel-good Thompson sampling — QuanquanGu · 2026-10-31