OpenAI's Jakub Pachocki: pretraining avalanche of advances conceivable by 2027
scaling01 · x · 2026-11-23
OpenAI chief scientist Jakub Pachocki shared internal-style metrics showing base models' next-token prediction (log perplexity / average cross-entropy loss) improving steadily. He said that "as soon as 2027 we could conceivably see a similar avalanche of fundamental pretraining advancements" akin to what's happened in mathematics. He cautioned that while AI will help with alignment and monitoring, that progress likely won't be fast enough to enable the full acceleration of pretraining, and called on AI labs to pace with "common limits" to understand capabilities first.
More from AGI Musings
- P=NP wouldn't break security — polynomial time isn't practical time — alex_peys · 2026-11-23
- Sam Altman calls OpenAI's pretraining transparency post its most important publish in a while — sama · 2026-11-23
- Blogger argues non-frontier AI labs like xAI and Mistral should pivot to inference providers — DionysianAgent · 2026-11-23
- "Many think they're centaurs; actually they're just being dragged by the horse" — Pseudomanifold · 2026-11-23
- Agent OS vision: MulticaAI dashboard shows Lead, Backend and QA agents collaborating on a work board — jiayuan_jy · 2026-11-23
- Shriram Krishnamurthi: 2026 could be an annus mirabilis for math but horribilis for mathematicians — IanArawjo · 2026-11-23