Forward-only pretraining speedup of 2.3x verified, inspired by interpretability essay
burny_tech · x · 2026-10-19
Developer @spiceylemonade turned ideas from Nick Cammarata's interpretability essay "Neural Networks Are Already Clean" into backprop-free pretraining speedups, achieving a verified 2.3x speedup.
Screenshots show a community leaderboard (NanoGPT-speedrun style) with two tracks racing val loss:
- Track 1: "Boost → Cut: stagewise readout training, then credit along strong routes" hit 3.945 val loss in just 8 minutes;
- Other wins include "associative credit memory as a prior" (3.998) and "reduce population to 16k" (3.991), with multiple contributors iterating.
Notable angle: interpretability insights being repurposed directly as training-efficiency optimizations rather than just for understanding models.
More from Research
- NTU's RoboFoundry evolves the robot software stack itself, lifting frozen GPT-5.5 from 56.9% to 72.7% — jiqizhixin · 2026-10-19
- Interp debate: Nick Cammarata says most neurons are explainable; leogao pushes back — burny_tech · 2026-10-19
- Baidu's Virtual Head Attention cuts KV cache 4-7x with zero custom kernels — burny_tech · 2026-10-19
- Classic DeepMind paper: the prefrontal cortex works as a meta-reinforcement learning system — burny_tech · 2026-10-19
- Steering side effects are real but neuron-specific, researcher argues — Sauers_ · 2026-10-19
- Steerable neurons: many steer output, few pin down a consistent persona — Sauers_ · 2026-10-19