Pedro Domingos: OpenAI admits pretraining has barely improved since 2022

pmddomingos · x · 2026-11-23

Noted researcher and ML textbook author Pedro Domingos says OpenAI has owned up to the fact that pretraining has barely improved since 2022. The accompanying chart shows base model log-perplexity staying in the 0.85-0.9 band from GPT-3.5 through the GPT-5 era, with only a late internal model dropping to 0.76 — implying raw pretraining gains have been far slower than product iteration suggests.

Related event: OpenAI Chief Scientist Predicts Pretraining Breakthrough as Soon as 2027(4 posts)→

Original post →

More from Models

Models channel →