Researcher: 'later transformer layers are useless' is an artifact of weak pretraining stacks
_aidan_clark_ · x · 2026-10-16
A concrete example of how interpretability findings depend on pretraining quality: lots of prior work claimed later transformer layers are largely useless, which is simply untrue on good pretraining stacks.
More from Research
- Free Textbook: Introduction to Machine Learning Covers the Math from Optimization to Generative Models — TheTuringPost · 2026-10-16
- Free ML math textbook: Laurent Younes' Introduction to Machine Learning covers Adam to generative models — TheTuringPost · 2026-10-16
- RFdiffusion3 published in Science, alongside Chaperone agent harness for bio-molecular design — anshulkundaje · 2026-10-16
- Microsoft paper: history-aware model routing lifts multi-turn agent task pass rate by 3.3 points — dair_ai · 2026-10-16
- Researcher: everyone should build a good eval at least once — bad evals are everywhere — gabriberton · 2026-10-16
- The square lattice connective constant remains unsolved — and might be ripe for AI — tak3sh8 · 2026-10-16