Looping Rewrite Scaling Exponents: 7.4B Model Matches GPT-3 13B With ~20x Less Compute
andrewgwils · x · 2026-09-17
A new paper from Andrew Gordon Wilson's group shows architectural interventions can modify pre-training scaling exponents: a 7.4B model-growth architecture via recursive depth (looping) matches GPT-3 13B on CORE with roughly 20x less compute, with efficiency gains that increase with scale. A simple boundary operator in vanilla transformers also helps, and looping regularizes in multi-epoch data-constrained settings.
More from Research
- General Reasoning explains how to train agent swarms with RL and latency pressure — rosstaylor90 · 2026-11-02
- Mathematician Tony Feng: current-paradigm LLMs lack the diversity of thought research needs — littmath · 2026-11-02
- Measure Theory vs Probability: 'You're Confusing the Finger With the Moon' — LucaAmb · 2026-11-02
- DStill Distills Multi-Step Denoising Into One Pass, 6.45x Speedup for Diffusion Speculative Decoding — LucaAmb · 2026-11-02
- Radial gate objective cuts diffusion transformer FID from 16.24 to 11.83 — Junha Park · 2026-11-02
- REA: async brain-cerebellum runtime for robots where policies never wait on the LLM — Pengfei Zhou · 2026-11-02