Explorative Modeling Unlocks a Third Pretraining Axis for Generative Models
illinois · hf · 2026-07-31
Current generative models typically factor the generation procedure into multiple stages, preventing true end-to-end training. This research introduces Explorative Modeling, which unlocks a new scaling dimension by factoring the training loop instead.
This method explores K candidate matches between model generations and data during each training step, training on the best match so predictions commit to modes rather than blurring them.
Key findings and results:
- Third Pretraining Axis: Beyond parameters and data, adding "exploration" monotonically improves performance. Gains increase with scale, improving FLOP efficiency by 4.1x and sample efficiency by 6.2x.
- End-to-End Generation: Enables end-to-end reconstructive generative modeling, matching diffusion performance on control tasks with 16-256x fewer inference steps.
More from Research
- Supabase Launches Evals to Benchmark AI Coding Agents on Real Tasks — tristanbob · 2026-08-01
- Stanford Researcher Shares DexterityGen and SPIDER for Cross-Embodiment Robot Skills — adamraudonis · 2026-08-01
- Viewpoint: LLMs Will Mechanically Increase the Rate of Scientific Gem Discovery — RexDouglass · 2026-08-01
- LabEvolver: Robots Become Better Scientists Without Weight Updates, Reaching 91% Success — imjustnewatai · 2026-08-01
- Genomic Intelligence Platform Rebuilt for Mobile Analysis — julia_kiseleva · 2026-08-01
- Raven: Linear-Time Sequence Model Achieves High-Recall via Sparse Memory Routing — bronzeagepapi · 2026-08-01