OpenAI floats perplexity-based safety bars; researchers urge a representative val set
_lewtun · x · 2026-11-23
OpenAI researcher merettm published "Perplexity and Intelligence", arguing for transparency into pretraining progress via common safety bars based on perplexity and other measurements, with further progress gated on our ability to understand AI behavior.
DimitrisPapail amplified the post, noting that releasing a representative validation set would be OpenAI's biggest contribution to open AI science — perplexity comparisons only work on a shared eval set. The take was also retweeted by Hugging Face's Lewis Tunstall.
More from Models
- OpenAI's Michelle Pokrass confirms GPT-4.1 wasn't a fresh pretrain, just a new mid-train — JasonBotterill · 2026-11-23
- 55-Page Riemann Hypothesis Proof Guide, De-Formalized from Lean with Gemini and GPT — chaumian · 2026-11-23
- Sarvam open-sources its latest 105B model; founder says scaling agentic RL 'is not for the faint hearted' — itsOmSarraf_ · 2026-11-23
- Gemini 4 Neon xhigh costs 8.6x more but adds just 1-4 points on most benchmarks — jeremyphoward · 2026-11-23
- Poland's 9B-parameter Thalamus Reasoning tops LongevityBench, beating 29 models — Dr_Singularity · 2026-11-23
- Dev: Gemini 4 Neon rewrote my training kernels and supervised runs; rumors point to bigger models — xennygrimmato_ · 2026-11-23