OpenAI's Jakub Pachocki: pretraining avalanche of advances conceivable by 2027

scaling01 · x · 2026-11-23

OpenAI chief scientist Jakub Pachocki shared internal-style metrics showing base models' next-token prediction (log perplexity / average cross-entropy loss) improving steadily. He said that "as soon as 2027 we could conceivably see a similar avalanche of fundamental pretraining advancements" akin to what's happened in mathematics. He cautioned that while AI will help with alignment and monitoring, that progress likely won't be fast enough to enable the full acceleration of pretraining, and called on AI labs to pace with "common limits" to understand capabilities first.

Original post →

More from AGI Musings

AGI Musings channel →