Perplexity Cross-Lab Comparisons Questioned: Tokenizer Differences Make Numbers Incomparable
davidmanheim · x · 2026-11-23
davidmanheim mocks a blog on perplexity as an AI progress metric for listing every limitation except the obvious one: different labs (e.g. Anthropic) use different tokenizers, so raw perplexity numbers aren't directly comparable.
The counter: 'anyone competent knows how to compare perplexity across tokenizers.' Another reply notes bits per byte fixes it — it's just a less catchy number. The attached blog excerpt does list real challenges: cross-lab dataset mismatch, detachment from actionable evaluations, chain-of-thought not reducing perplexity, and an unknown loss floor.
Related event: Perplexity Cross-Model Comparison Dispute Prompts Response from Boaz Barak(5 posts)→
More from Research
- Fine-tuned Gemma 4 E4B drives 100 villagers offline in a rumor-spread simulation game — AlternativeEnough352 · 2026-11-23
- LingSante-3.1-air-VL tops medical imaging evals across derm, labs and X-rays — Dr_Alex_Crimi · 2026-11-23
- Mathematician praises Lean community's soundness work, admits skepticism is easier than building — mboehme_ · 2026-11-23
- New proposal: mandatory compute disclosures plus 15% capabilities cap could delay intelligence explosion a year — sjgadler · 2026-11-23
- Blackbench ranks decision models by how much money they win at blackjack — kukicolaa · 2026-11-23
- Prune, train, upcycle: one model that scales with your RAM — minhnguyen · 2026-11-23