Explorative Modeling Unlocks a Third Pretraining Axis for Generative Models
illinois · hf · 2026-07-31
Current generative models typically factor the generation procedure into multiple stages, preventing true end-to-end training. This research introduces Explorative Modeling, which unlocks a new scaling dimension by factoring the training loop instead.
This method explores K candidate matches between model generations and data during each training step, training on the best match so predictions commit to modes rather than blurring them.
Key findings and results:
- Third Pretraining Axis: Beyond parameters and data, adding "exploration" monotonically improves performance. Gains increase with scale, improving FLOP efficiency by 4.1x and sample efficiency by 6.2x.
- End-to-End Generation: Enables end-to-end reconstructive generative modeling, matching diffusion performance on control tasks with 16-256x fewer inference steps.
Related event: Explorative Modeling: Third Pretraining Axis, Up to 6x Sampling Efficiency(12 posts)→
More from Research
- Transforming quadruped robot runs on real hardware: per-pose RL policies in Isaac switched on morph — _Stocko_ · 2026-10-16
- KinoGen synthesizes robot loco-manipulation reference motions directly via kinematically grounded video generation — ChongZzZhang · 2026-10-16
- Science Robotics paper introduces F³, predictive event-based vision beyond optical flow — CSProfKGD · 2026-10-16
- VideoGen-Agent: An 8B Multimodal Agent Turns Video Generation Into a Tool-Use Loop — jiqizhixin · 2026-10-16
- Comprehension Audits: New Paper Proposes a Gate for AI-Written Code at Frontier Labs — ronbodkin · 2026-10-16
- Not Boring #215: Cammarata's Clean Networks Claim, Tesla's Flying Roadster, Tau Cut 73% — Not Boring (Packy McCormick) · 2026-10-16