Hugging Face Ships tokenizers v1 with Up to 30x Speedup

On September 21, Hugging Face released the first release candidate of tokenizers v1, the library's first major version, announced by core member Lysandre. The version is a full performance rewrite, improving encode/decode speed 3x to 30x over v0.23, with tokenization up to 30x faster, competitive with the best tokenization implementations in the ecosystem.

Confirmed

• v1 is a Rust implementation focused on performance bottlenecks; tokenization was historically not a training bottleneck, but as models get faster, datasets grow, and concurrent serving increases, CPU-side tokenization can leave GPUs idle waiting for data (@ariG23498 relayed the team's motivation)
• Better scaling across threads with significantly improved latency (@eliebakouch)
• Training functionality split into an independently opt-in crate, with heavily optimized algorithms and benchmarks (@pcuenq)
• Integration into transformers coming soon (@pcuenq)

Why it matters

Tokenization has long been treated as a secondary step in training and inference pipelines, but it has become a real bottleneck in large-scale training, high-concurrency inference, and long-input processing. If v1 delivers its promised 3x–30x speedups and lands smoothly in transformers, it will directly reduce GPU idle time and cut serving latency, with real throughput gains across the Hugging Face ecosystem.

2026-09-21 ~ 2026-09-21 · 7 related posts

• Hugging Face Ships tokenizers v1, Often Tens of Times Faster Than v0.23 — art_zucker · 2026-09-21
• [source] Hugging Face releases Tokenizers v1 RC with up to 30x faster tokenization — LysandreJik · 2026-09-21
• Hugging Face ships tokenizers v1 with multilingual bindings, multi-thread scaling, smaller package — Disastrous-Work-1632 · 2026-09-21

4 near-duplicate retellings: pcuenq · eliebakouch · ariG23498 · ariG23498