Tokenizer trilemma: BPB metric measures coding length and nothing else, thread argues
RylanSchaeffer · x · 2026-10-03
Rylan Schaeffer closes an 11-part thread on Bits-per-Byte (BPB) as an LLM evaluation metric with two predictions for the field:
- BPB tells us about coding length and nothing else; the community has given it meaning beyond what it carries.
- A tin-foil-hat take: intelligence isn't about compression but about discriminating correct from plausible-but-incorrect continuations.
Earlier in the thread he proves a tokenizer trilemma: of three reasonable desiderata, a metric can satisfy at most two — choosing a metric implicitly chooses what research can be done.
Related event: Researcher Debunks Tokenizer Myths, Proves Tokenizer Trilemma(3 posts)→
More from Research
- NVIDIA's SoL-Pi uses AI agents to cut coding agent API costs by ~50% vs Codex — dr_cintas · 2026-10-03
- MIT's Cathy Wu boosts RL training efficiency 30x to design smarter transportation systems — MIT News AI · 2026-10-03
- Economist calls for mirror life regulations; researchers say the threat is wildly premature — anshulkundaje · 2026-10-03
- Benchmarking long-range ML interatomic potentials on SrTiO3 defect energetics — rbhar90 · 2026-10-03
- Meta AI's RL-XAR fixes AI slop via expert-aligned rubrics, plus 4 more papers explained — idanbeck · 2026-10-03
- Neuralink pretrained AI models on 50,000 hours of unlabeled brain data, betting on the scaling playbook — Dr_Singularity · 2026-10-03