OpenAI Embraces Perplexity Transparency, Critics Urge Releasing Held-Out Eval Set

scaling01 · x · 2026-11-23

OpenAI researcher merettm announced greater transparency into pretraining progress in "Perplexity and Intelligence", arguing for shared safety bars based on perplexity and gating further progress on understanding AI behavior. Reposter oneillc urged OpenAI to go further and release its held-out perplexity evaluation set, claiming it would instantly leapfrog Anthropic in safety contributions. Jakub's quoted view: cross-lab comparison is hard due to differing eval sets and data mixtures, but transferable comparison methods are possible if transparency becomes a shared goal.

Related event: OpenAI Discloses Pretraining Perplexity Data, Proposes Industry Safety Thresholds(18 posts)→

Original post →

More from Models

Models channel →