H-Net vs BPE scaling fight: critic's fit puts crossover at ~4e24 FLOPs
teortaxesTex · x · 2026-10-30
- Albert Gu's original post: the new H-Net (hierarchical networks learning tokenization end-to-end from raw bytes) shows up to 3-5x efficiency gains in controlled scaling-law experiments and matches a strong open-source baseline at larger scale — a real step toward the bitter lesson of learning all abstractions from raw data.
- szhaai's rebuttal: fitting L = E + A·C^(−α) to the blog's data gives BPE a slightly better exponent (0.1366 vs 0.1338); H-Net's lead shrinks with compute, with crossover projected around 4e24 FLOPs — within modern pretraining budgets. Suspected causes: large-vocab BPE baselines dominated by last-layer compute, or suboptimal vocab size.
- Third-party fit (chart): across 1e18–1e20 FLOPs, H-Net's compute advantage shrinks 4.7% per 10x compute (1.37x → 1.22x), but 95% CIs include zero (P=0.92/0.90 for exponent gap), so the result is suggestive, not conclusive.
More from Models
- Microsoft's BitNet brings 1.58-bit weights to embeddings, 2.28x FP16 throughput on CPU — techNmak · 2026-10-30
- Rumored DeepSeek Flash-Cyber benchmarks leak, claiming ExploitBench 96.8 — teortaxesTex · 2026-10-30
- Gemini scheduled tasks fail three days straight, blame the user, and still burn quota — Rude-Requirement5642 · 2026-10-30
- DeepSeek's cyber model gates access with export-control and sanctions questionnaires, leaked frontend code shows — teortaxesTex · 2026-10-30
- Anthropic's Claude now self-assesses conversations to flag misuse others miss — elonmusk · 2026-10-30
- Cost-based pricing makes tokenmaxxing always profitable — but may dull model progress — menhguin · 2026-10-30