SentencePiece adds Auto-character Coverage as a clean alternative to Byte-Level BPE
prajdabre · x · 2026-09-12
SentencePiece 0.2.3 introduces an "Auto-character Coverage" mode that globally optimizes the character vs. subword budget within a fixed vocabulary via mathematical optimization, avoiding the malformed UTF-8 byte fragments typical of Byte-Level BPE while achieving comparable or better compression. The docs include benchmarks across 13 languages on a 390MB Wikipedia corpus and algorithms for BPE and Unigram models.
More from Research
- WISD: Distill Training Signal from Weaker Quantized Copies of Reasoning Models — jet-ai · 2026-10-31
- Albert Gu: H-Net learns to segment Chinese characters from raw bytes mid-training — niloofar_mire · 2026-10-31
- LLM Acknowledgments in Bio Preprints Hit 17% in Three Years, Near Specialized AI Tools — teortaxesTex · 2026-10-31
- CS theorist explains the two meanings of 'exponential time' behind new 3-SAT result — aran_nayebi · 2026-10-31
- StudentBench author: AI tutors boosting human intelligence is the use-case to hill-climb — gharik · 2026-10-31
- Spatial Hub aggregates ~5000 public COLMAP-compatible spatial datasets for world models — janusch_patas · 2026-10-31