Larger tokenizers suffer more multi-epoch regret, small vocabs survive repeats
SonglinYang4 · x · 2026-11-04
Wenhao Chai's new experiment blog finds that while n-gram models dominate recent pre-training speedruns, they fall apart in multi-epoch setups. Training a fixed amount of text for 8 epochs, larger tokenizers incur more multi-epoch regret — the extra loss from repeating data instead of training on unique tokens. Small vocabularies survive better.
The post walks through byte-level BPE mechanics (merging frequent pairs from 256 bytes, embedding and softmax rows per token), demos encoding with Llama 3's Marin 128K tokenizer, and publishes code, training curves, and the full experiment log.
More from Models
- LLMs struggle with "surgicality": legal redlines leave unnecessary ink on the page — graceisford · 2026-11-04
- Frontier models race sailboats in a sim: Argon wins, Opus DNFs, Qwen runs aground — shekitup · 2026-11-04
- Domingos: Gemini itself agrees CoT and RL are a wasteful detour for reasoning — pmddomingos · 2026-11-04
- Legora CTO on frontier models that can understand and modify their own weights — garrytan · 2026-11-04
- LoL architecture reorders its own forward pass at runtime, mimicking algorithmic composition — micahgoldblum · 2026-11-04
- Grok Bot weekly quota 30% used within 9 hours of reset, user asks for bigger pool — nima_owji · 2026-11-04