OpenAI opens pretraining data: time horizon scales exponentially with perplexity
danielrock · x · 2026-11-23
- OpenAI's "Perplexity and Intelligence" blog discloses pretraining progress data and argues for common safety bars based on perplexity, gating further progress on understanding AI behavior.
- Basil Halperin's analysis shows METR 50% time horizon scales exponentially with pretraining loss: each 0.032 drop in log-perplexity doubles the horizon (≈0.025 in the GPT-3.5→GPT-4o era).
- Reasoning models (o1/o3/GPT-5.x) appear as a one-time jump in this relationship with slope largely unchanged; the chart includes an unreleased GPT-6 Astra point extrapolating to 256+ hour horizons.
More from Models
- MIT prof: model capabilities follow a scaling law, with a threshold for open math discovery already crossed — ProfBuehlerMIT · 2026-11-23
- 'You're Routed to Opus 4.6': Matthew Berman's Model-Routing Surprise Moment — MatthewBerman · 2026-11-23
- Reddit Hands-on With inkling-mini-2: Coherent and Careful, but Overly Cautious — Express_Quail_1493 · 2026-11-23
- Matt Wolfe: Google Is Back With Its New AI Model — Weekly AI News Roundup — Matt Wolfe · 2026-11-23
- Google's AI Overview triggers on single letter but not complex queries — VeredShwartz · 2026-11-23
- Voyager releases open-source shot-reader-8b multimodal model on Hugging Face — ravisparikh · 2026-11-23