Pedro Domingos: OpenAI admits pretraining has barely improved since 2022
pmddomingos · x · 2026-11-23
Noted researcher and ML textbook author Pedro Domingos says OpenAI has owned up to the fact that pretraining has barely improved since 2022. The accompanying chart shows base model log-perplexity staying in the 0.85-0.9 band from GPT-3.5 through the GPT-5 era, with only a late internal model dropping to 0.76 — implying raw pretraining gains have been far slower than product iteration suggests.
Related event: OpenAI Chief Scientist Predicts Pretraining Breakthrough as Soon as 2027(4 posts)→
More from Models
- Inkling-2-Mini hits 10.04x on KernelBench-Mega locally, beating Grok 4.7 and GPT-6.1 — ziqiao_ma · 2026-11-23
- Baidu open-sources Unlimited-OCR: 3B model reads 100-page PDFs in one shot — udmrzn · 2026-11-23
- Vocab Surgery: editing 10 dictionary words drops a model's refusal rate to near zero — morgymcg · 2026-11-23
- Perplexity across tokenizers: researcher pushes back on Anthropic comparison controversy — quantum_geoff · 2026-11-23
- Thinking Machines releases Inkling 2 Small, scoring 31 on AA Intelligence Index — _Hao_Zhu · 2026-11-23
- Leaked OpenAI loss ladder: unknown capabilities await at 0.68-0.7 — tallmetommy · 2026-11-23