Researcher: BPE tokenizer wins by inertia, alternatives face challenges

AI researcher kalomaze argues that BPE's dominance is largely inertia, noting that a simple linear projection over fixed-size UTF-8 bytes works surprisingly well, though tokenizer-free approaches still struggle with needing a fixed-size computation unit, a key weakness of H-net.

2026-10-29 ~ 2026-10-29 · 2 related posts

• kalomaze: dumb UTF-8 linear projection baselines work, BPE wins only by inertia — kalomaze · 2026-10-29
• kalomaze: fixed-size compute units plagued H-net, and BPE wins only by inertia — kalomaze · 2026-10-29