Researcher pre-registers prediction that new tokenizer's BPB gains are misleading
RylanSchaeffer · x · 2026-10-29
Rylan Schaeffer publicly pre-registers predictions after requesting checkpoints: the new tokenizer will show lower BPB on both correct and incorrect benchmark choices while downstream performance stays roughly the same. His proposed mechanism: it isn't learning to separate correct from incorrect continuations, just carving text into smaller, easier-to-predict pieces — so BPB measured only on correct continuations is misleading.
More from Models
- Claim: Chinese labs now distilling from OpenAI's Astra after Anthropic blockade — bindureddy · 2026-10-29
- "Desired oververbosity 5": Claude spots OpenAI distillation fingerprints in DeepSeek reasoning traces — zeeshanp_ · 2026-10-29
- Qwen4 leak points to 1M-token context window, Qwen4-27B expected tomorrow — brainExploded99 · 2026-10-29
- Muse-Glimmer flops in hands-on test: user gives up at 40K tokens, switches to DeepSeek — techmago · 2026-10-29
- Pedro Domingos mocks NYT: the brain runs one dimension per synapse — a trillion — pmddomingos · 2026-10-29
- Gemini 4 Carbon posts impressive Terminal-Bench-Science score, per early reports — geoffwolfe · 2026-10-29