Was the Perplexity Comparison Dishonest? Anthropic's Tokenizer Made It Unfair
davidmanheim · x · 2026-11-23
davidmanheim presses on motive: did the perplexity-comparison blog not know Anthropic's tokenizer made the comparison unfair, or did it just not want to suggest a less dishonest comparator? The reply: bits per byte fixes that, it's just a less catchy number.
The exchange exposes a real issue in industry perplexity benchmarks: each lab reports numbers in its own tokenizer, inherently favoring itself; bits per byte is the fairer unit but lacks punch.
Related event: Perplexity Cross-Model Comparison Dispute Prompts Response from Boaz Barak(5 posts)→
More from Research
- Fine-tuned Gemma 4 E4B drives 100 villagers offline in a rumor-spread simulation game — AlternativeEnough352 · 2026-11-23
- LingSante-3.1-air-VL tops medical imaging evals across derm, labs and X-rays — Dr_Alex_Crimi · 2026-11-23
- Mathematician praises Lean community's soundness work, admits skepticism is easier than building — mboehme_ · 2026-11-23
- New proposal: mandatory compute disclosures plus 15% capabilities cap could delay intelligence explosion a year — sjgadler · 2026-11-23
- Blackbench ranks decision models by how much money they win at blackjack — kukicolaa · 2026-11-23
- Prune, train, upcycle: one model that scales with your RAM — minhnguyen · 2026-11-23