kalomaze: fixed-size compute units plagued H-net, and BPE wins only by inertia
kalomaze · x · 2026-10-29
In a tokenizer-free training discussion, kalomaze argues the core problem is wanting a fixed-size unit of computation — H-net's worst flaw was that this unit changed across training, making fair comparisons at equal bpb + FLOPS hard.
He also shares a counterintuitive practical finding: a "stupid" baseline of linearly projecting fixed-size UTF-8 bits with no dedicated embedding table has worked well for him, suggesting BPE's dominance is mostly inertia rather than fundamental merit.
Related event: Researcher: BPE tokenizer wins by inertia, alternatives face challenges(2 posts)→
More from Research
- LoopVL: 1B-parameter VLM reuses 32 layers across 128 calls, beats 4B baselines on 5/8 benchmarks — jiqizhixin · 2026-10-29
- Geoffrey Irving puts >90% on unbounded-depth CSETH, unsure about finite depth — geoffreyirving · 2026-10-29
- davidad: lead poisoning may explain the 1971–2012 Great Stagnation, checked with GPT-4 — davidad · 2026-10-29
- New AI security benchmarks from Meta collaborators urge community to double down on robustness — maksym_andr · 2026-10-29
- Arena Releases Alignment Index, Arguing Alignment Is a Function of Model Capability — arena · 2026-10-29
- Researchers Push Back on AI Audit of Saez-Zucman Papers Over Missing Prompts — felpix_ · 2026-10-29