Stanford paper: BPB isn't tokenizer-agnostic, so don't judge models by it alone
rohanpaul_ai · x · 2026-11-06
A new Stanford paper shows bits-per-byte (BPB), the standard 'tokenizer-agnostic' metric, is biased: untrained models guessing uniformly still score different BPB by tokenizer (English peaks near a 24K vocabulary) while accuracy stays at chance, and on real models BPB correlates with task scores only between +0.02 and −0.30. The 'unigram-normalized loss' metric's reported 0.989 correlation with BPB drops to 0.057 once a confounder is fixed, and the two metrics pick different vocabularies. The authors formalize a tokenizer trilemma: no metric can simultaneously score only the true distribution best, give identical predictions identical scores, and tie all untrained models. Practical takeaway: never treat lower BPB as proof of a better model—check task accuracy on the same text before picking a vocabulary.
Related event: Stanford paper: bits-per-byte is not tokenizer-agnostic(3 posts)→
More from Research
- Frank Nielsen's taxonomy maps every distance and divergence from Euclidean to quantum geometry — pickover · 2026-11-06
- Anthropic report's key estimates are "guesses" — framing hides massive uncertainty, researcher says — daveholtz · 2026-11-06
- Project AURORA: a big-bet AI Cancer Co-scientist initiative launches with CRUK and NCI backing — zakkohane · 2026-11-06
- PNAS paper: kids explore farther than RL agents, driven by empowerment not info gain — _akpiper · 2026-11-06
- Google open-sources Agile Deliberation, an AI framework that turns fuzzy classification rules into explicit ones — aigclink · 2026-11-06
- ETH RSL's ACP humanoid: reference-free contact planning runs onboard at 50Hz — ChongZzZhang · 2026-11-06