Researchers Push Back on Perplexity Safety Bars: Data Mixture Bias Undermines Cross-Lab Comparisons

eliebakouch · x · 2026-11-23

HF researcher Elie Bakouch responds to OpenAI's perplexity-based safety bar proposal, arguing PPL is largely a data-mixture issue: a 100%-code pretraining mix would trivially lower PPL on a code held-out set. He proposes a standardized "pretraining index" of curated document sets, and guesses top labs' mixtures are similar enough for rough comparability.

aidanclark adds the hardest problem: cross-lab equivalence, since methods improving PPL may be fundamentally equivalent to ones that only improve PT evals — he is skeptical of absolute PPL thresholds, though rates of improvement may be meaningful.

Original post →

More from Safety

Safety channel →