Meta FAIR's large-scale study distills byte models to rival token-based LLMs
mgostIH · x · 2026-10-24
Meta FAIR and University of Washington researchers (including Luke Zettlemoyer and Mike Lewis) released "Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models." The paper introduces two methods to convert token logits to byte logits (approximate Marginalize-It and exact End-Of-Token) and runs the first large-scale overtraining study of decoder-only dense transformers varying both tokenization scheme (Tokens/Bytes/Bytes with <eot>) and training objective (distillation vs cross-entropy), training 1B-parameter models on up to a trillion bytes. Across eight benchmarks spanning multiple-choice QA, generation, and machine translation, it examines when byte models can close the gap with token-based models.
Related event: Meta Paper Shows Byte-Level Models Can Beat Token Models After Distillation(3 posts)→
More from Research
- CoRL paper: flow-matching VLAs can run 1-step Euler inference with no retraining — kaixhin · 2026-10-26
- CoVer at EMNLP 2026: Training Code LLMs to Generate and Self-Verify Simultaneously — algo_diver · 2026-10-26
- Robotic dexterity should be judged on speed, force control and tool use, not copying humans — DJiafei · 2026-10-26
- Turing-complete malware can run in PyTorch's backward pass via a zero-valued loss — retr0jirachi · 2026-10-26
- Nathan Lambert's first open models lecture: distillation, risks and the ecosystem explained — natolambert · 2026-10-26
- Search researcher on MRL debate: losses are tolerable but benchmarks are too small with stale QRELs — srchvrs · 2026-10-26