Exploration as a Third Pretraining Axis: Major Boost in Data and Compute Efficiency
kastnerkyle · x · 2026-08-02
Alexi Glad introduces "Explorative Modeling," discovering a third axis for pretraining beyond parameters and data: exploration.
Key findings include:
- Scaling gains: The benefits of exploration grow with scale, improving from 7% to 36% as data scales, and 13% to 23% as parameters scale. Gains double at 3× compute.
- Massive efficiency improvements: Adding exploration to near-SOTA baselines improves data efficiency by 6.2×, FLOP efficiency by 4.1×, and parameter efficiency by 47%.
- Strong generation performance: Achieves a near-SOTA unguided FID of 1.43 on ImageNet.
The method allows trading training compute for generalization and scales end-to-end generation capabilities. In its simplest form, it is implemented via a for loop.
More from Research
- Simplex Diffusion Models paper talk by EPFL/DeepMind's Justin Deschenaux on Oct 19 — ArnaudDoucet1 · 2026-10-16
- Meituan LongCat team's Data World synthesizes data environments to train data agents — meituan-longcat · 2026-10-16
- FDE-Gym benchmark shows huge gap on real AI delivery tasks: top model scores 0.563 — DeepLethe · 2026-10-16
- ForcingBooster halves computation of autoregressive video diffusion with patch-wise denoising — VAST-AI · 2026-10-16
- Baidu open-sources FlashMask: up to 1.55x faster than FlashAttention-4 for long-context training — BaiduResearch · 2026-10-16
- SemEval-2027 Task 9 opens: multimodal news framing analysis across 4 languages — preslav_nakov · 2026-10-16