Looped transformers study: 7.4B growth model matches GPT-3 13B with 20x less compute
burny_tech · x · 2026-09-19
arXiv paper 2609.19107 shows architectural interventions can change pre-training scaling exponents:
- Weight sharing is not compute-optimal on fresh data, costing a constant 1.06–1.16x compute at every scale; its benefit only appears in data-constrained, multi-epoch training.
- The gains of looping come from the norm + input re-injection boundary operator and growing depth mid-training — both improve the scaling exponent without weight sharing.
- Model growth yields the biggest exponent shifts: a 7.4B growth architecture matches GPT-3 13B on CORE with roughly 20x less compute, with efficiency gains that increase with scale.
- In multi-epoch settings, looping regularizes, and it is compute-optimal to increase loop count with scale.
More from Infra
- Malaysia poised to become a compute superpower with both US and Chinese chips by next year — basedjensen · 2026-11-04
- ByteDance details USDB storage system: 85.8% lower write latency for recommender sequences — _reachsumit · 2026-11-04
- Oracle cancels $7B chip leasing deal with Tencent, FT reports — SumitGup · 2026-11-04
- Inference engine shrunk to 1023 bytes, still runs SmolLM2-135M — cneuralnetwork · 2026-11-04
- Musk says space data centers are 'looking pretty appealing right now' — elonmusk · 2026-11-04
- Data centers are turning to old jet engines as the wait for power grows — IndependentCoast3974 · 2026-11-04