Perplexity Cross-Lab Comparisons Questioned: Tokenizer Differences Make Numbers Incomparable

davidmanheim · x · 2026-11-23

davidmanheim mocks a blog on perplexity as an AI progress metric for listing every limitation except the obvious one: different labs (e.g. Anthropic) use different tokenizers, so raw perplexity numbers aren't directly comparable.

The counter: 'anyone competent knows how to compare perplexity across tokenizers.' Another reply notes bits per byte fixes it — it's just a less catchy number. The attached blog excerpt does list real challenges: cross-lab dataset mismatch, detachment from actionable evaluations, chain-of-thought not reducing perplexity, and an unknown loss floor.

Related event: Perplexity Cross-Model Comparison Dispute Prompts Response from Boaz Barak(5 posts)→

Original post →

More from Research

Research channel →