Albert Gu: H-Net learns to segment Chinese characters from raw bytes mid-training
niloofar_mire · x · 2026-10-31
Albert Gu shared that H-Net's blog and animation didn't show the training dynamics for multi-byte languages like CJK. Chinese characters take up 3 bytes each, and chunking them correctly isn't trivial when learning end-to-end from bytes: early in training the model incorrectly sub-divides characters, but eventually figures it out — a prominent issue in the original H-Net that even BPE gets wrong on rare characters, with examples in the blog.
H-Net embodies the 'bitter lesson' bet that intelligence should come from simple, general modules learning abstractions directly from raw data; the new version shows 3-5x efficiency gains in controlled scaling law experiments.
More from Research
- Tversky branch inside RNN cells: half the depth matches full xLSTM with 2x faster training — stanfordnlp · 2026-10-31
- AI pushes circuit lower bound from 3.1n to 3.5n as UCLA-backed challenge aims at P vs NP — soumitrashukla9 · 2026-10-31
- Qwenputer: a full RISC-V CPU inside Qwen3.5-4B weights, running C in forward passes — armand_ruiz · 2026-10-31
- What is work for? Cross-national survey of 8,234 adults on meaning in the AI age — soumitrashukla9 · 2026-10-31
- Paper: benchmark difficulty coverage is skewed, leaderboard conclusions need caution — Miserable-Dare5090 · 2026-10-31
- AISTATS 2027 workshop proposals due Nov 6 after one-week extension — qberthet · 2026-10-31