kalomaze: dumb UTF-8 linear projection baselines work, BPE wins only by inertia
kalomaze · x · 2026-10-29
Researcher kalomaze argues that BPE's dominance in tokenization is owed to inertia: he has gotten solid results with "stupid" baselines that just do a linear projection of fixed-size UTF-8 bits with no dedicated embedding table at all.
Related event: Researcher: BPE tokenizer wins by inertia, alternatives face challenges(2 posts)→
More from Research
- BlackboxNLP kicks off at EMNLP2026 Budapest, opening keynote on how LLMs read — gsarti_ · 2026-10-29
- QUAN paper: real-valued quaternion convolution nets SOTA in oriented detection — Bryce Grant · 2026-10-29
- Tim Vieira Explains Conditional Poisson Sampling with Verified Identities and Interactive Visuals — xtimv · 2026-10-29
- Reward-model ensembles from different pretraining seeds resist reward hacking better — vxnuaj · 2026-10-29
- Google DeepMind engineer shows five CUDA optimizations that speed up 3D Gaussian Splatting by 10.5x — janusch_patas · 2026-10-29
- AI Audit Finds 2,035 Errors in 138 Papers by Economists Saez and Zucman — sebkrier · 2026-10-29