Hugging Face Ships tokenizers v1 RC with Up to 30x Speedup

Hugging Face released the first release candidate of tokenizers v1 on September 21, announced by core member Lysandre. It is a full Rust rewrite delivering 3–30x faster encoding/decoding versus v0.23, with tokenization up to 30x faster—competitive with the best implementations in the ecosystem—and is set to be integrated into transformers.

Confirmed

v1 is Rust-based and targets performance bottlenecks; tokenization used to rarely be the bottleneck, but faster models, larger datasets, and higher-concurrency serving can leave GPUs idle waiting on CPU tokenization (@ariG23498)
Stronger cross-thread scaling with significantly improved latency (@eliebakouch)
Training split into an optional standalone crate with many optimized algorithms and benchmarks (@pcuenq)
Bindings for more languages and a much smaller package size (@Disastrous-Work-1632)
Per @jasonkneen (repost), the rewrite achieves a 15.9x speedup without changing any token IDs, via SIMD splitters replacing the regex engine, caching for repeated words, and zero-allocation merged loops
Upcoming integration into transformers (@pcuenq)

Why it matters

Tokenization was long treated as a minor part of training and inference pipelines, but at scale it has become a real bottleneck. If v1 delivers its 3–30x speedup and lands in transformers, it reduces GPU idle time and serving latency; the unchanged token IDs mean users can upgrade at no cost, benefiting throughput across the Hugging Face ecosystem.

2026-09-21 ~ 2026-09-21 · 8 related posts

Primary sources

Hugging Face releases Tokenizers v1 RC with up to 30x faster tokenization — LysandreJik · 2026-09-21 22:02 (UTC+08:00)
Hugging Face ships tokenizers v1: 3-30x faster encoding and decoding — pcuenq · 2026-09-21 22:17 (UTC+08:00)
Hugging Face ships tokenizers v1 with multilingual bindings, multi-thread scaling, smaller package — Disastrous-Work-1632 · 2026-09-21 23:10 (UTC+08:00)

Hugging Face Ships tokenizers v1, Often Tens of Times Faster Than v0.23 — art_zucker · 2026-09-21
[source] Hugging Face releases Tokenizers v1 RC with up to 30x faster tokenization — LysandreJik · 2026-09-21
[source] Hugging Face ships tokenizers v1 with multilingual bindings, multi-thread scaling, smaller package — Disastrous-Work-1632 · 2026-09-21
Hugging Face makes tokenizers 15.9x faster without changing a single token ID — jasonkneen · 2026-09-21

4 near-duplicate retellings: pcuenq · eliebakouch · ariG23498 · ariG23498