Byte H-Net matches SmolLM3-3B with over 10x fewer training tokens
xennygrimmato_ · x · 2026-10-29
The shared chart (Figure 7) compares Byte-level 2-stage H-Net against SmolLM3-3B (BPE Transformer) under compute- and data-matched pretraining:
- On the six-task average score, Byte H-Net reaches in 1T tokens what SmolLM3-3B needs 11T tokens to achieve — over 10x fewer training tokens
- Both curves converge to a similar 64% average, showing the tokenizer-free byte-level H-Net architecture is dramatically more data-efficient than BPE-based Transformers
One of the strongest data-efficiency results yet for the tokenizer-free line of research.
More from Research
- ESM2 embeddings plus a linked tripartite graph for exploring TF–motif data — jmschreiber91 · 2026-10-29
- Interactive filters for TF–motif interaction data: family, source, and confidence — jmschreiber91 · 2026-10-29
- Preprint: Human peer reviewers caught only 27% of major errors, all 15 AI models beat them (53-96%) — anshulkundaje · 2026-10-29
- New paper rules out Hamiltonian Phase States as a route to quantum-only Microcrypt — StefanoGogioso · 2026-10-29
- MSR AI for Science team hiring postdoc to generate molecular data for Skala — FrankNoeBerlin · 2026-10-29
- ML-enhanced resonant soft X-ray scattering enables sub-monolayer tracking of buried-interface molecular states — carbon-lab · 2026-10-29