H-Net learns dynamic tokenization for code, showing 3-5x efficiency gains
gharik · x · 2026-10-29
Tri Dao highlights a striking H-Net result: the model learns to dynamically tokenize variable names in coding data as separate tokens instead of relying on BPE, scaling much better on code.
In the quoted post, Albert Gu frames this as a step toward the bitter lesson: intelligence should come from simple, general building blocks that learn their own abstractions from raw data. Byte-level language modeling is a microcosm — if LLMs can't learn end-to-end from bytes, how will they handle data types we understand even less?
- H-Nets were introduced last year as a first step
- The new version shows up to 3-5x efficiency gains in controlled scaling-law experiments
- At larger scale it matches open-source BPE-based models
Related event: Cartesia's H-Net Ditches Tokenizers for Raw-Byte Language Modeling(6 posts)→
More from Models
- Developer replaces Siri with local LFM3-Audio-2B, cutting home control latency from 3s to 0.54s — gregbarbosa · 2026-10-29
- 48h of AI chaos: Anthropic IPO, GPT-6.1 Terra, Genie 4, and a Gamma 5 agent demo — dr_cintas · 2026-10-29
- FrontierSWE leaderboard: GPT-6 Astra tops at 65.5%, Gemini 4 Carbon scores 61.2% at 1/16th the cost — scaling01 · 2026-10-29
- DeepSeek Flash fingerprint change hints at migration to a new GPU cluster — teortaxesTex · 2026-10-29
- Google admits human review error in Gemini policy flags, reverses account strikes — RexDouglass · 2026-10-29
- AA-PersonalityBench: Meta's Muse Models Rank Most Friendly, Nemotron 3 Ultra Scores Just 1 — ArtificialAnlys · 2026-10-29