Meta paper: distilled byte models may beat Llama 3.2-1B with 6x less data
DotaMate · reddit · 2026-10-24
Meta FAIR and University of Washington's paper "Breaking the Token Ceiling" trains 1B models that read raw bytes instead of tokens, distilling knowledge from token-based teachers (Llama 3 8B) via two logit-conversion methods (Marginalize-It and End-Of-Token).
- With little training, token models win easily; but they plateau while byte models keep improving.
- Extrapolated curves predict distilled EOT-1B beats Llama 3.2-1B by up to 4% on averaged downstream tasks — but that's a prediction; at actually trained points token models still lead.
- Byte models match token-model quality with 1/6 the training data, use a 256-byte vocabulary (no top-k truncation, 5x cheaper logit storage).
- Tradeoffs: 5x more compute steps per text, teacher still needs tokens, untested at scale.
If it holds, this matters a lot for on-device small models and could fix tokenizer quirks like letter miscounting.
Related event: Meta Paper Shows Byte-Level Models Can Beat Token Models After Distillation(3 posts)→
More from Models
- Google AI Pro/Ultra subscribers now get $10-$100 monthly Cloud credits for AI Studio and Vertex AI — testingcatalog · 2026-10-26
- Claude's answer to what you'll regret not owning in 10 years: your own AI — alex_verem · 2026-10-26
- China Telecom's XingChen lab open-sources Xing4.0-29B-Intent, a 29B MoE with 4B activated params — FellMentKE · 2026-10-26
- Deleted vLLM PR Hints Qwen4-27B Uses 4-Way Parallel Residual Streams (Unconfirmed) — Ok-Shower7286 · 2026-10-26
- a16z: OpenRouter token demand up ~27x in a year, open-weight models now serve 72% — TheZachMueller · 2026-10-26
- Running a 125B model on 8GB VRAM with Strata: 900 t/s prompt processing, 30 t/s generation — BleynSpecnaz · 2026-10-26