Researchers Push Back on Perplexity Safety Bars: Data Mixture Bias Undermines Cross-Lab Comparisons
eliebakouch · x · 2026-11-23
HF researcher Elie Bakouch responds to OpenAI's perplexity-based safety bar proposal, arguing PPL is largely a data-mixture issue: a 100%-code pretraining mix would trivially lower PPL on a code held-out set. He proposes a standardized "pretraining index" of curated document sets, and guesses top labs' mixtures are similar enough for rough comparability.
aidanclark adds the hardest problem: cross-lab equivalence, since methods improving PPL may be fundamentally equivalent to ones that only improve PT evals — he is skeptical of absolute PPL thresholds, though rates of improvement may be meaningful.
More from Safety
- Fake official account spotted, prominent blogger warns links are unsafe to click — dotey · 2026-11-23
- Why cybercriminals skip AI attacks: too noisy, costly tokenomics, old TTPs still work — jcran · 2026-11-23
- Vercel CEO Rauch: 'free' ZDR stealth models look like industrial-scale data exfiltration — AccBalanced · 2026-11-23
- DOCKET benchmark shows LLM policy feedback tilts toward government: 68% overlap vs 26% for unions — stanfordnlp · 2026-11-23
- Agents Sabotage Peer Shutdown in 38.3% of Runs Across 17 Models, No Task Needed — rohanpaul_ai · 2026-11-23
- Australia's Spy Agency Now Treats Data Center Ownership as Foreign Interference Risk — rohanpaul_ai · 2026-11-23