H-Net learns tokenization on the fly from raw bytes, matching models with 10x less data and compute
MikePFrank · x · 2026-10-29
Albert Gu's team unveiled a new version of H-Net, a hierarchical architecture that learns end-to-end from raw bytes without a fixed tokenizer.
- Learns the basic structure of raw data on the fly: after seeing code a few times, it tokenizes unseen programming languages
- Shows up to 3-5x efficiency gains in controlled scaling law experiments, and converges to the same performance as conventional models with 10x less data and compute
- Gu frames it as a real step toward the bitter lesson ideal—intelligence from simple, general building blocks that learn their own abstractions; if LLMs can't learn from bytes, how will they handle the world's data we understand even less?
The quoted post is from Mamba co-author Albert Gu, who endorsed the work.
Related event: Cartesia's New H-Net Learns from Raw Bytes, 3-5x Efficiency Gains(12 posts)→
More from Research
- Hexels: JHU's Hexagonal Primitives Cut VRAM 5.5x While Beating 3DGS by 4.96dB — ssh4net · 2026-10-29
- 3,957-task study finds no reliable way to combine human and AI oversight yet — sethlazar · 2026-10-29
- New preprint measures 'articulation gaps': how much knowledge in humans and AI is tacit — sanmikoyejo · 2026-10-29
- Peer Review Is the Least Articulable Field: Five Findings on Tacit Knowledge — sanmikoyejo · 2026-10-29
- New Prompt Optimization Beats GEPA by ~14% and Estimates Performance Bounds — sanmikoyejo · 2026-10-29
- Borrowing Capture-Recapture from Ecology to Bound Rubric Performance — sanmikoyejo · 2026-10-29