Researcher pre-registers predictions on tokenizer's effect before model release
RylanSchaeffer · x · 2026-10-29
Rylan Schaeffer says he has requested the model checkpoints and publicly pre-registered his predictions: the new tokenizer achieves lower bpb on both correct and incorrect answer choices for downstream benchmarks, and downstream performance will be roughly unchanged. The mechanism: it has not learned to separate correct from incorrect answers. The pre-registration approach offers a testable way to judge whether tokenizer changes actually improve capability.
More from Research
- Perfect hashing hits 1.3ns lookups, ~12x faster than std::unordered_map — arpit_bhayani · 2026-10-29
- Qualia Beats Flora and GPT-6 Astra Across Three Robot-Data Benchmarks — lukas_m_ziegler · 2026-10-29
- FlashAttention vs standard attention in one chart: the bottleneck is data movement, not compute — cneuralnetwork · 2026-10-29
- LLM agents forecast with $500 budgets; GPT splurges, Grok/Kimi/GLM spend nothing — karinanguyen · 2026-10-29
- DeepMind's Jon Barron: every softmax and cross-entropy implementation has two subtle bugs — jon_barron · 2026-10-29
- H-Net update shows 3-5x efficiency gains in raw-byte language modeling — omarsar0 · 2026-10-29