H-Net tokenizer vs BPE: 2.4x fewer tokens on short lyrics benchmark
mgostIH · x · 2026-10-29
mgostIH amplifies a tokenizer test of a 'bitter lesson edition' H-Net checkpoint: on short lyrics, it produces 2.4x fewer tokens than BPE. Screenshots show BPE shattering repetitive onomatopoeia into many fragments while H-Net merges long repeated spans into single tokens. The author is taking requests for more texts to tokenize.
More from Research
- Dev ditches unanimous-vote stopping rule for a judge in multi-AI debate project — lluisantoni · 2026-10-30
- Google's AI Search Uses Your Knowledge Graph Facts Without Linking, RESONEO Research Finds — dejanseo · 2026-10-30
- Raon-1-mini-preview runs natively on hipfire but 200x slower prefill than llama.cpp, patch released — Queasy_Asparagus69 · 2026-10-30
- TEM: Transformer Energy-based Models scale beyond diffusion without guidance — du_yilun · 2026-10-30
- NVIDIA's AgentBaton cuts agent API costs 65.9% with a single model handoff — dair_ai · 2026-10-30
- Stanford releases RT-J, a System 1 model predicting structured data in one forward pass — jure · 2026-10-30