Boaz Barak defends internal perplexity metric: not directly transferable across models
boazbaraktcs · x · 2026-11-23
Boaz Barak pushed back on criticism of his team's blog post metric, clarifying that it is an internal measure explicitly flagged as not directly transferable across models, with tokenizer differences being a minor units issue. He stressed that the phenomenon they care about is inherently compression, which can be measured across models with the right methods. The dispute matters because the metric was floated as a basis for a common standard for banning further training — critics like David Manheim argue per-byte perplexity should have been reported instead.
Related event: Perplexity Cross-Model Comparison Dispute Prompts Response from Boaz Barak(5 posts)→
More from Safety
- Mark Chen calls on frontier labs to share safety cases and a common progress metric — eliebakouch · 2026-11-23
- New proposal: mandatory compute disclosures plus 15% capabilities cap could delay intelligence explosion a year — sjgadler · 2026-11-23
- OpenAI's chief scientist on how to put a speed limit on AI development — TechCrunch AI · 2026-11-23
- Fireworks AI breach hits 150k Uber drivers; US tightens compute-access rules — DavidLinthicum · 2026-11-23
- Fake official account spotted, prominent blogger warns links are unsafe to click — dotey · 2026-11-23
- Why cybercriminals skip AI attacks: too noisy, costly tokenomics, old TTPs still work — jcran · 2026-11-23