Meta paper: byte-level models overtake token models at scale, up to 4% ahead

omarsar0 · x · 2026-09-14

• A Meta paper shows byte-level models start behind token models but overtake them as compute grows, demonstrated on distilled 1B models trained on up to 1 trillion bytes.
• The distillation converts a token teacher's logits into byte logits, either approximately (Marginalize-It) or exactly (End-Of-Token).
• Key findings: token models lead at low compute but plateau; byte models reach a higher ceiling, with fitted scaling laws predicting End-Of-Token ends up to 4% ahead of the distilled token model.
• Efficiency gains: byte models match the distilled token model with one-sixth of the training data, and a 256-entry vocabulary cuts teacher-logit storage to about one-fifth.

Original post →

More from Research

• ELLIS x UniReps talk: why distinct learning processes yield similar representations — ClementineDomi6 · 2026-10-30
• Physicist calls OpenAI's Navier-Stokes solution physically meaningless despite loophole win — GaryMarcus · 2026-10-30
• Anthropic paper: 0.1% pretraining poison plants RCE backdoor in 4B CLI agent — omarsar0 · 2026-10-30
• CopeNLU presents 7 papers at EMNLP 2026 on bias, alignment, explainability — IAugenstein · 2026-10-30
• Two-stage 3D ResNet pipeline detects neonatal brain pathologies from MRI with 0.919 AUC — Dr_Alex_Crimi · 2026-10-30
• 40 years of putting new neurons into the CNS: why so slow, and can AI accelerate it? — patrickmineault · 2026-10-30

Research channel →