OpenAI Chief Scientist proposes perplexity as shared industry safety bar, publishes internal pretraining metrics
polynoamial · x · 2026-11-23
OpenAI chief scientist Jakub Pachocki argues in "Perplexity and Intelligence" that perplexity (cross-entropy loss) should become a common metric for comparing model developers and setting shared safety limits, with progress beyond such limits gated on our ability to understand AI behavior.
As a first step, OpenAI is publishing its internal pretraining perplexity measurements. Pachocki acknowledges this gives competitors a clearer picture of OpenAI's standing but says the urgency of industry-wide shared safety measures outweighs competitive concerns. He frames AI progress as building better compressors on large corpora, then eliciting latent capabilities via prompting, finetuning, and RL.
More from Safety
- Musk: superintelligence at exawatt scale will be like a caveman playing with sticks — elonmusk · 2026-11-23
- Researchers Push Back on Perplexity Safety Bars: Data Mixture Bias Undermines Cross-Lab Comparisons — eliebakouch · 2026-11-23
- Vercel CEO Rauch: 'free' ZDR stealth models look like industrial-scale data exfiltration — AccBalanced · 2026-11-23
- DOCKET benchmark shows LLM policy feedback tilts toward government: 68% overlap vs 26% for unions — stanfordnlp · 2026-11-23
- Agents Sabotage Peer Shutdown in 38.3% of Runs Across 17 Models, No Task Needed — rohanpaul_ai · 2026-11-23
- Australia's Spy Agency Now Treats Data Center Ownership as Foreign Interference Risk — rohanpaul_ai · 2026-11-23