Exploration as a Third Pretraining Axis: Major Boost in Data and Compute Efficiency
kastnerkyle · x · 2026-08-02
Alexi Glad introduces "Explorative Modeling," discovering a third axis for pretraining beyond parameters and data: exploration.
Key findings include:
- Scaling gains: The benefits of exploration grow with scale, improving from 7% to 36% as data scales, and 13% to 23% as parameters scale. Gains double at 3× compute.
- Massive efficiency improvements: Adding exploration to near-SOTA baselines improves data efficiency by 6.2×, FLOP efficiency by 4.1×, and parameter efficiency by 47%.
- Strong generation performance: Achieves a near-SOTA unguided FID of 1.43 on ImageNet.
The method allows trading training compute for generalization and scales end-to-end generation capabilities. In its simplest form, it is implemented via a for loop.
More from Research
- Retriever: A Framework for Asynchronous, Closed-Loop Robot Agents — ZeYanjie · 2026-08-24
- Converting GMMs ↔ PEFs for fast KLD approximation — FrnkNlsn · 2026-08-24
- Netflix details its production LLM judge: hundreds of thousands of recommendations scored weekly — omarsar0 · 2026-08-24
- Nature Comment: Provenance, not interpretability, grounds trust in autonomous science — gabepgomes · 2026-08-24
- New Architecture RHEA: Train 1B Model on 8GB VRAM — zemondza · 2026-08-24
- Trained two 16M-param models to do generative CAD with real physics — debreuil · 2026-08-24