Nature paper converts subword LLMs to byte-level with under 1% of pretraining cost, yielding Bolmo
Justgototheeffinmoon · reddit · 2026-10-22
A Nature paper published October 7, 2026 introduces "byteification": converting existing subword language models into byte-level ones for 49.1 billion tokens total—under 1% of a typical pretraining budget (a 9.8B-token first stage with the global model frozen, then 39.3B tokens in stage two).
Key facts:
- Led by Benjamin Minixhofer, with co-authors including Luke Zettlemoyer, Noah A. Smith, Luca Soldaini and Valentin Hofmann.
- Applied to four open models—OLMo 3 7B, OLMo 2 1B, Qwen3 8B Base, Llama 3 8B—yielding Bolmo 7B, Bolmo 1B, Bwen 8B and Blama 8B; parameter counts barely moved (+4.5% / -0.7%).
- Bolmo 7B delivers a +16.5% absolute improvement in STEM tasks over the prior byte-level baseline BLT 7B, while keeping general-benchmark performance close to the subword source.
- Motivation: subword tokenization obscures fine-grained information problematic for scientific data like code and biological sequences; byte-level models previously lagged in performance.
- Two technical choices: non-causal boundary prediction using one byte of future context, and doubling the byte vocabulary from 256 to 512 by fusing boundary symbols with bytes. The models run at practical inference speeds and remain compatible with existing serving stacks.
More from Research
- We need an L0.5 for continual learning: agent state stacks up in 4 layers — willcb · 2026-10-22
- JOLT trains the teacher too, with penalties when its predictions diverge from the student — perplexity_ai · 2026-10-22
- Perplexity's JOLT matches GRPO with 6.5x fewer training tokens, JOLT+ wins on all five benchmarks — perplexity_ai · 2026-10-22
- Parallel Decoding Distillation Adopted by Alibaba, NVIDIA, Amazon Weeks After Release — ArashVahdat · 2026-10-22
- RFD4-Proteina hits 3.5% DNA-binder pass rate vs 0.8% for BoltzGen — AllThingsApx · 2026-10-22
- RFD4-Proteina passes computational checks on all 77 small-molecule targets — AllThingsApx · 2026-10-22