SentencePiece Lite ships: 50KB binary, 20-30x faster tokenization for edge devices
heiga_zen · x · 2026-09-16
Taku Kudo released SentencePiece Lite Runtime, a zero-dependency C++ tokenizer (50KB binary, <2k LOC) that runs Unigram/BPE directly on memory-mapped FlatBuffers models, achieving zero-copy startup, 20-30x faster throughput and far fewer heap allocations. It retains normalization, byte fallback, stochastic sampling and offset tracking with 100% compatibility, ships as a single .cc amalgamation, and will become SentencePiece's next-gen core engine.
More from Infra
- YuE2 music sampling only hits ~7 tokens/s on RX 9070 despite mostly idle VRAM — Prestigious-Kick7291 · 2026-09-16
- As LMs mature, the work reduces to two things: data and infra — saurabh_shah2 · 2026-09-16
- Leaked 10,000-word Huawei memo: become the NVIDIA for any LLM, pivot around Ascend 950 — pstAsiatech · 2026-09-16
- Doubao AI phone tops 360K pre-orders; RTX 5090 scalped at $9,500 — 创业邦 · 2026-09-16
- Running Qwen 27B and DeepSeek v4 Flash together on one heterogeneous machine — samsja19 · 2026-09-16
- JPMorgan sees 25M+ GPU/ASIC shipments by 2028, ASICs dominate — a 'narrative violation' — bookwormengr · 2026-09-16