New Paper: Looping With Model Growth Bends Scaling Laws, Compute Gains Compound
akbirthko · x · 2026-09-17
The authors' new paper challenges the assumption that architectural changes only give constant-factor gains while pretraining progress comes mostly from data. They find that model growth, looping, and boundary operators yield compute multipliers over standard transformers that grow exponentially with each order of magnitude of compute:
- 1.55x at 1e20 FLOPs, projected 2.7x at 1e25
- Matches GPT-3 13B on CORE with 20x less compute
The core claim: the scaling exponent itself can be improved by architecture, with gains that compound with compute rather than staying constant.
More from Research
- Detecting covert instruction-following via internal representations parallels J-space work — dioscuri · 2026-11-02
- Xiaomi MiMo's CodeSpecWeave trains coding agents on interdependent specs for long-horizon coding — XiaomiMiMo · 2026-11-02
- Mistral Large 4 writes a quantum chemistry SCF solver from scratch with one short prompt — MistralAI · 2026-11-02
- Hunting for the most concrete "curing cancer with AI" roadmap of 2025 — iskander · 2026-11-02
- New theory paper proves linear decay is the unique minimax-optimal stepsize schedule for SGD — konstmish · 2026-11-02
- Karpathy's nanochat: the best ChatGPT $100 can buy, near 60k GitHub stars — karpathy · 2026-11-02