OpenAI publishes internal pretraining perplexity curve, proposes perplexity as shared safety bar

merettm · x · 2026-11-23

OpenAI Chief Scientist Jakub Pachocki published "Perplexity and Intelligence," disclosing internal pretraining perplexity measurements for the first time and arguing that perplexity (equivalently, cross-entropy loss) should serve as a comparable metric across model developers and inform shared safety standards. His thesis: modern AI is largely about training compressors on huge corpora, so tracking compression progress is key to understanding AI's arc, and progress beyond perplexity thresholds should be gated on our ability to understand model behavior. The accompanying chart shows base-model log-perplexity falling from 0.99 (GPT-3.5) to 0.76 (GPT-6.2 Astra), with capability thresholds (fluent conversation, thinking before answering, useful coder, cybersecurity expert, solved a Millennium Prize problem) mapped to perplexity levels — plus an unreleased internal model already below the Millennium Prize threshold.

Original post →

More from Safety

Safety channel →