OpenAI floats perplexity-based safety bars; researchers urge a representative val set

_lewtun · x · 2026-11-23

OpenAI researcher merettm published "Perplexity and Intelligence", arguing for transparency into pretraining progress via common safety bars based on perplexity and other measurements, with further progress gated on our ability to understand AI behavior.

DimitrisPapail amplified the post, noting that releasing a representative validation set would be OpenAI's biggest contribution to open AI science — perplexity comparisons only work on a shared eval set. The take was also retweeted by Hugging Face's Lewis Tunstall.

Original post →

More from Models

Models channel →