SentencePiece adds Auto-character Coverage as a clean alternative to Byte-Level BPE
prajdabre · x · 2026-09-12
SentencePiece 0.2.3 introduces an "Auto-character Coverage" mode that globally optimizes the character vs. subword budget within a fixed vocabulary via mathematical optimization, avoiding the malformed UTF-8 byte fragments typical of Byte-Level BPE while achieving comparable or better compression. The docs include benchmarks across 13 languages on a 390MB Wikipedia corpus and algorithms for BPE and Unigram models.
More from Research
- Columbia to host symposium on AI, genomics and community science in biogeography — sarameghanbeery · 2026-09-12
- Japan's JST-MEXT to host AI for Science 2026 symposium featuring Google DeepMind — heiga_zen · 2026-09-12
- Fruit Fly Brain Simulation Solves Rubik's Cube, Igniting Consciousness Debate — sebkrier · 2026-09-12
- 25 Fields Medal winners warn AI's goals are 'severely misaligned' with mathematics — The Decoder · 2026-09-12
- LuxoBench: A New AI Benchmark Tasks Models With Building Real Electromechanical Devices — vincent_koc · 2026-09-12
- Q2D-Web benchmark debuts: 70K agent queries to evaluate retrievers across 190M web docs — antoine_chaffin · 2026-09-12