OpenAI publishes internal pretraining perplexity curve, proposes perplexity as shared safety bar
merettm · x · 2026-11-23
OpenAI Chief Scientist Jakub Pachocki published "Perplexity and Intelligence," disclosing internal pretraining perplexity measurements for the first time and arguing that perplexity (equivalently, cross-entropy loss) should serve as a comparable metric across model developers and inform shared safety standards. His thesis: modern AI is largely about training compressors on huge corpora, so tracking compression progress is key to understanding AI's arc, and progress beyond perplexity thresholds should be gated on our ability to understand model behavior. The accompanying chart shows base-model log-perplexity falling from 0.99 (GPT-3.5) to 0.76 (GPT-6.2 Astra), with capability thresholds (fluent conversation, thinking before answering, useful coder, cybersecurity expert, solved a Millennium Prize problem) mapped to perplexity levels — plus an unreleased internal model already below the Millennium Prize threshold.
More from Safety
- Dario Amodei tells Congress open-source AI is heading toward dangerous, uncontrollable territory — ylecun · 2026-11-23
- Anthropic's on-site power is 80% simple-cycle gas; health damages estimated at $2-3B per year — petrusenko_max · 2026-11-23
- Writers Push Google for Gemini 18+ Age Verification and Content Toggles to Stop Exodus to Grok — Hikkynure · 2026-11-23
- CMU's ExploitBench ranks LLM cyber agents on a capability ladder, not pass/fail — goyal__pramod · 2026-11-23
- Windows 11 caught running hidden telemetry that tracks which AI coding agents you use — cyb3rops · 2026-11-23
- AI can't do the impossible: exposed IT weaknesses were always there, argues researcher — JHochderffer · 2026-11-23