Stanford paper: bits-per-byte is not tokenizer-agnostic

A Stanford preprint by Rylan Schaeffer and Sanmi Koyejo shows bits-per-byte (BPB) is not truly tokenizer-agnostic—untrained models that guess uniformly still yield different BPB across tokenizers—urging caution against judging models by a single metric.

2026-11-06 ~ 2026-11-06 · 3 related posts

• New preprint 'Tokenizer Myths and Where to Find Them' busts tokenization folklore — RylanSchaeffer · 2026-11-06
• Stanford paper: BPB isn't tokenizer-agnostic, so don't judge models by it alone — rohanpaul_ai · 2026-11-06
• Tokenizer Myths and Where To Find Them: Stanford paper debunks three BPB myths — rohanpaul_ai · 2026-11-06