Meta paper ditches tokenizers: distilled 1B byte models outperform token-based peers
mtizard · x · 2026-10-23
Meta FAIR and University of Washington released "Breaking the Token Ceiling," a large-scale study arguing LLMs can drop tokenizers entirely and process raw bytes over a 256-value vocabulary.
- Two methods convert token logits to byte logits: approximate Marginalize-It and exact End-Of-Token.
- They swept 1B-parameter dense transformers across tokenization scheme and training objective (distillation vs cross-entropy), scaling up to 1 trillion bytes of data over 8 benchmarks.
- Distilled End-Of-Token Bytes-1B beats Token-1B in the low-FLOP regime and reaches a higher performance ceiling; extrapolated scaling laws show up to 4% higher downstream accuracy with 6x better data efficiency.
- Bonus: skipping top-k logit truncation cuts logit storage to roughly one-fifth; EOT-1B is projected to asymptotically surpass Llama 3.2-1B (by up to 6.5%), Gemma-3-1B-pt (8.1%), and Gemma 2B (2.1%).
More from Models
- OpenAI blocks manual reset with 13% usage left: 'no reset needed right now' — AccomplishedSugar490 · 2026-10-23
- Meta's 816B Muse Spark Open scores 39, losing to 180B Qwen3.8-Flash-Next at 40 — BTA_Labs · 2026-10-23
- Gemini refuses to generate image in Docs, blames document size limit — viksit · 2026-10-23
- Dev jokes his true calling is telling everyone to try the latest DeepSeek — intellectronica · 2026-10-23
- SubQ Snap 1.1 ties for first on Artificial Analysis Decision Index with 384K context — ArtificialAnlys · 2026-10-23
- PolyAI unveils Dialog-RSN-1: audio-native voice model answering in under 300ms — matthen2 · 2026-10-23