Boaz Barak defends internal perplexity metric: not directly transferable across models

boazbaraktcs · x · 2026-11-23

Boaz Barak pushed back on criticism of his team's blog post metric, clarifying that it is an internal measure explicitly flagged as not directly transferable across models, with tokenizer differences being a minor units issue. He stressed that the phenomenon they care about is inherently compression, which can be measured across models with the right methods. The dispute matters because the metric was floated as a basis for a common standard for banning further training — critics like David Manheim argue per-byte perplexity should have been reported instead.

Related event: Perplexity Cross-Model Comparison Dispute Prompts Response from Boaz Barak(5 posts)→

Original post →

More from Safety

Safety channel →