Perplexity across tokenizers: researcher pushes back on Anthropic comparison controversy
quantum_geoff · x · 2026-11-23
In an ongoing dispute over cross-tokenizer perplexity comparisons, quantumgeoff argues that anyone competent already knows how to compare perplexity across tokenizers (and how not to), so the controversy is overstated: the subtle issues are real, but tokenizer unfairness isn't one of them. Others note bits per byte solves the problem, just with a less catchy number.
More from Models
- Pedro Domingos: OpenAI admits pretraining has barely improved since 2022 — pmddomingos · 2026-11-23
- Inkling-2-Mini hits 10.04x on KernelBench-Mega locally, beating Grok 4.7 and GPT-6.1 — ziqiao_ma · 2026-11-23
- Baidu open-sources Unlimited-OCR: 3B model reads 100-page PDFs in one shot — udmrzn · 2026-11-23
- Vocab Surgery: editing 10 dictionary words drops a model's refusal rate to near zero — morgymcg · 2026-11-23
- Thinking Machines releases Inkling 2 Small, scoring 31 on AA Intelligence Index — _Hao_Zhu · 2026-11-23
- Leaked OpenAI loss ladder: unknown capabilities await at 0.68-0.7 — tallmetommy · 2026-11-23