Explorative Modeling: Discovering a Third Pretraining Axis Beyond Parameters and Data
burny_tech · x · 2026-08-03
Alexi Glad introduced Exploration, a third pretraining axis beyond parameters and data.
- Core Finding: Scaling exploration monotonically improves existing models across images, video, and language, unlocking end-to-end generation. In its simplest form, it's just a for loop.
- Scaling Gains: Benefits grow with scale, jumping from 7% to 36% as data scales, and 13% to 23% as parameters scale. Gains double at 3× the compute.
- Efficiency: Adding exploration to near-SOTA baselines improves data efficiency by 6.2×, FLOP efficiency by 4.1×, and parameter efficiency by 47%, hitting a near-SOTA 1.43 unguided FID on ImageNet.
Developer @cgarciae88 tested the method and confirmed it works, though they noted a slight reduction in diversity. In 2D examples, the distribution matches the overall shape but lacks smoothness in certain areas.
More from Research
- A missing parameter in scaling laws: H, the human effort behind learning signals — lateinteraction · 2026-10-16
- One ancient fold, many jobs: how the immunoglobulin domain evolved into antibodies, MHC and titin — arjunrajlab · 2026-10-16
- Researcher Argues Conferences Should Desk-Reject Far More Papers as Flooding Strains Reviews — boknilev · 2026-10-16
- Universal forgery attack breaks Chinese PQC signature Origami using only the public key — matthew_d_green · 2026-10-16
- How do you attribute agent gains: model training or infrastructure? — jrxna · 2026-10-16
- First-ever paper on multi-agent swarm safety lands NeurIPS spotlight — LauraRuis · 2026-10-16