Albert Gu's new H-Net hits 3-5x efficiency gains learning directly from raw bytes
stochasticchasm · x · 2026-10-29
Albert Gu (Mamba co-author) unveils a new version of Hierarchical Networks (H-Nets), arguing intelligence should come from simple, general building blocks that learn their own abstractions from raw data — the bitter lesson realized.
Key points:
- If LLMs can't learn end-to-end from bytes, how will they handle the world's data we understand even less? Byte-level language modeling is the microcosm of this ideal.
- The new H-Net achieves up to 3-5x efficiency gains in controlled scaling-law experiments and at larger scale matches open tokenizer-based baselines.
- It exhibits emergent "in-context tokenization": dynamically learning to segment novel words in-context (shown vs. a standard tokenizer on made-up words like borvexila/mipralune).
- Jonathan Chang and others amplified the result, calling in-context tokenization "super cool."
Related event: Albert Gu's H-Net boosts byte-level modeling efficiency 3-5x(3 posts)→
More from Models
- Practitioner: No serious lab actually trains on the test set today — nrehiew_ · 2026-10-29
- Benchmaxxing is mostly training focus, not cheating, argues nrehiew_ on FrontierCode — nrehiew_ · 2026-10-29
- Rumor: OpenAI pushes big release to tomorrow, possibly due to Gemini 4 Carbon — mark_k · 2026-10-29
- Gemini 4.0 Carbon's security praised, but mocked for losing the "felony bench" — vitaliychiley · 2026-10-29
- Google hits frontier on private benchmarks yet stock plunges 8% after hours — bookwormengr · 2026-10-29
- Dev praises gpt-6-luna's price-performance, calls opus-5.5 best for implementation — emax · 2026-10-29