Nearly 600 runs show full BPTT beats truncated backprop in looped transformers
Xuyuan Xiong · hf · 2026-10-28
A HKU MMLab & Yale study systematically examines truncation in looped transformer training. Varying model size, training tokens, and forward/backward loop counts across nearly 600 runs, the authors fit a unified scaling law via "effective parameters".
Key findings:
- Despite compute savings, the fitted law favors full BPTT when loop counts are jointly optimized;
- In matched-compute runs, full BPTT through fewer loops consistently beats truncated settings;
- Optimal loop count and token budget grow with compute at fixed model size; with adjustable model size, two fully-trained loops are optimal.
More from Research
- 2020 physics paper links learning rate, Hessian and gradient noise via fluctuation-dissipation theorem — burny_tech · 2026-10-28
- EMNLP 2026 Hosts Tutorial on Failure Modes in Agentic AI, Mapping Four Stages of Agent Building — wzenus · 2026-10-28
- Jeremy Avigad: the AI debate belongs at the center with us, not the AI — Machine Learning Street Talk · 2026-10-28
- Meituan LongCat's FVQ makes discrete visual tokens match continuous representations — meituan-longcat · 2026-10-28
- Single neuron steering works far worse than contrastive-pair control vectors in practice — burny_tech · 2026-10-28
- Effective learning rate explains peak-then-decay schedules as bias-variance optimum — burny_tech · 2026-10-28