Researcher pushes back on internal perplexity metric: report per-byte or admit tokenizer gaps
davidmanheim · x · 2026-11-23
David Manheim replies to Boaz Barak criticizing the blog's evaluation metric: since different tokenizers make perplexity non-comparable across models, they should have reported per-byte perplexity, or at least listed the issue among known caveats—especially as the metric was touted as a basis for a common standard for banning further training. Barak's side responds that the blog clearly framed it as an internal metric, not directly transferable, and that tokenizer differences are a relatively minor units issue; the phenomenon they care about is inherently compression, which can be measured across models.
Related event: Perplexity Cross-Model Comparison Dispute Prompts Response from Boaz Barak(5 posts)→
More from Research
- Fine-tuned Gemma 4 E4B drives 100 villagers offline in a rumor-spread simulation game — AlternativeEnough352 · 2026-11-23
- LingSante-3.1-air-VL tops medical imaging evals across derm, labs and X-rays — Dr_Alex_Crimi · 2026-11-23
- Mathematician praises Lean community's soundness work, admits skepticism is easier than building — mboehme_ · 2026-11-23
- New proposal: mandatory compute disclosures plus 15% capabilities cap could delay intelligence explosion a year — sjgadler · 2026-11-23
- Blackbench ranks decision models by how much money they win at blackjack — kukicolaa · 2026-11-23
- Prune, train, upcycle: one model that scales with your RAM — minhnguyen · 2026-11-23