OpenAI publishes internal perplexity data, proposes it as shared industry safety metric
btibor91 · x · 2026-11-23
OpenAI chief scientist Jakub Pachocki argues for perplexity (cross-entropy loss) as a shared metric to compare model developers and inform safety pacing, publishing OpenAI's internal pretraining measurements for the first time.
- GPT-3.5 sits at log-perplexity 0.99, GPT-4 at 0.91; capabilities in cybersecurity and math emerge below 0.8
- Further transformative capabilities are expected around the 0.68–0.7 window
- OpenAI proposes common limits for developers until AI monitoring reaches a concrete milestone, acknowledging the competitive cost of transparency
More from Safety
- Sam Altman Details Common Limits Proposal: Shared Norms and Concrete Milestones Before Crossing Development Limits — sama · 2026-11-23
- OpenAI Publishes Internal Pretraining Perplexity Data, Calls for Industry-Wide Safety Limits — sama · 2026-11-23
- Gary Marcus: AI-Powered Cyberattacks Are Exploding, Bridgewater Pegs Destabilizing Attack at 1-in-3 Odds — GaryMarcus · 2026-11-23
- Dario Amodei tells Congress open-source AI is heading toward dangerous, uncontrollable territory — ylecun · 2026-11-23
- Anthropic's on-site power is 80% simple-cycle gas; health damages estimated at $2-3B per year — petrusenko_max · 2026-11-23
- Writers Push Google for Gemini 18+ Age Verification and Content Toggles to Stop Exodus to Grok — Hikkynure · 2026-11-23