New paper: looping with model growth improves scaling exponent, matches GPT-3 with 20x less compute
burny_tech · x · 2026-09-19
A new paper challenges the assumption that architectural changes yield only constant-factor gains: looping with model growth improves the scaling exponent, producing compute multipliers that grow exponentially with each OOM of compute.
- 1.55x at 1e20 FLOPs, projected 2.7x at 1e25
- Matches GPT-3 13B on CORE with 20x less compute
Steve Hsu frames the RSI angle: if usable computational depth drives the scaling exponent, a model that inspects its residual stream, detects vanishing later-block contributions, and proposes mid-training loop growth plus a refined boundary operator is already performing narrow recursive self-improvement — cheap to code and test, compounding with each cycle. He links it to DeepSeek's mHC, which treats the residual stream as the redesign object via doubly stochastic mixing matrices on the Birkhoff polytope.
More from Research
- Zoom's 176-setting ablation study reveals which coding harness components actually matter — dair_ai · 2026-09-19
- Google has two internal models beating median superforecasters on ForecastBench — Tolopono · 2026-09-19
- NVIDIA's SoL-Pi auto-evolves agent harnesses, cutting tokens ~50% and API costs ~33% with no quality loss — omarsar0 · 2026-09-19
- Betting war: 1:3 odds P vs NP or Riemann gets solved within a year — morqon · 2026-09-19
- JEPA-Anything: one predictive framework spanning vision, biology, weather and more — rbhar90 · 2026-09-19
- Tsinghua's C2C Lets LLMs Skip Text and Merge KV-Caches Directly, 2.5x Faster with +14.2% Accuracy — anselm · 2026-09-19