Perplexity is a flawed cross-lab capability bound: tokenizer granularity favors OpenAI
davidmanheim · x · 2026-11-23
David Manheim argues that perplexity is a flawed metric to standardize on for cross-lab capability bounds: Anthropic's more fine-grained tokenizer yields inherently lower per-token perplexity, so any shared perplexity-based limit would let OpenAI push much further. A subtle but consequential measurement bias in lab safety agreement design.
More from Models
- Google's BuggyMLE injects bugs into ML repos, dropping top model solve rate from 95% to 9% — omarsar0 · 2026-11-23
- OpenAI chart reveals unreleased GPT-6 Astra models; Domingos questions billions for <0.3 bits — pmddomingos · 2026-11-23
- Replit agents speed up NumPy 18.9x — but only Claude's team scales — amasad · 2026-11-23
- Gemini 4 Hydrogen at $1.50/M Input Tokens Scores 51, Beating Argon's 50 — dl_weekly · 2026-11-23
- Base log perplexity is an extremely strong predictor of Epoch and OpenAI ECI, researchers note — stochasticchasm · 2026-11-23
- Redis creator antirez: AI is already great at math and programming, blaming the tool is the real problem — antirez · 2026-11-23