Researcher pushes back on internal perplexity metric: report per-byte or admit tokenizer gaps

davidmanheim · x · 2026-11-23

David Manheim replies to Boaz Barak criticizing the blog's evaluation metric: since different tokenizers make perplexity non-comparable across models, they should have reported per-byte perplexity, or at least listed the issue among known caveats—especially as the metric was touted as a basis for a common standard for banning further training. Barak's side responds that the blog clearly framed it as an internal metric, not directly transferable, and that tokenizer differences are a relatively minor units issue; the phenomenon they care about is inherently compression, which can be measured across models.

Related event: Perplexity Cross-Model Comparison Dispute Prompts Response from Boaz Barak(5 posts)→

Original post →

More from Research

Research channel →