Perplexity across tokenizers: researcher pushes back on Anthropic comparison controversy

quantum_geoff · x · 2026-11-23

In an ongoing dispute over cross-tokenizer perplexity comparisons, quantumgeoff argues that anyone competent already knows how to compare perplexity across tokenizers (and how not to), so the controversy is overstated: the subtle issues are real, but tokenizer unfairness isn't one of them. Others note bits per byte solves the problem, just with a less catchy number.

Original post →

More from Models

Models channel →