Hugging Face Tokenizers v1 Push: BERT Tokenizer Gets Another 5x Speedup

Hugging Face is reviving the tokenizers library and sprinting toward v1.0.0. With optimizations from Simon Brandeis, the WordPiece tokenizer used by bert-base-uncased can now run about 5.7x faster on top of v1's existing gains—roughly 50x faster overall compared to the old v0.23. The optimization is currently in a PR and has not yet been merged. This has direct value for large-scale text preprocessing and embedded model inference.

Confirmed

The tokenizers v1 rewrite has already dramatically boosted tokenization speed: the classic BPE tokenizer encodes about 30x faster than v0.23 (@tomaarsen)
The WordPiece tokenizer used by bert-base-uncased is about 8.6x faster in v1, making it one of the relatively slower ones (@tomaarsen)
Simon Brandeis wrote a detailed post on further optimizations for BERT-style tokenizers: tricks like skipping ASCII normalization deliver roughly a 5.7x speedup, about 50x total over v0.23, currently at the PR stage (@tomaarsen, shared by @artz Zucker)
Sentence Transformers maintainer @tomaarsen noted that this tokenization rewrite can significantly accelerate inference for the most widely used Sentence Transformer models, with static embedding models benefiting the most

Why it matters

Tokenization is often an overlooked bottleneck in NLP pipelines; speedups on the order of 30–50x have real impact for both massive corpus preprocessing and real-time inference
WordPiece/BERT-style models are among the most widely used tokenization schemes in the industry, so targeted optimizations have broad coverage
Downstream libraries like Sentence Transformers can benefit directly, with especially notable gains for static embedding use cases

2026-10-15 ~ 2026-10-15 · 5 related posts

Primary sources

Tokenmaxxing BERT: how skipping ASCII normalization made tokenization 50x faster — tomaarsen · 2026-10-15 17:15 (UTC+08:00)
Hugging Face tokenizers: another ~5x speedup for BERT tokenizers on top of v1 — tomaarsen · 2026-10-15 17:15 (UTC+08:00)
Tokenization refactor speeds up Sentence Transformers, biggest gains for static embedding models — tomaarsen · 2026-10-15 17:15 (UTC+08:00)

[source] Tokenmaxxing BERT: how skipping ASCII normalization made tokenization 50x faster — tomaarsen · 2026-10-15
BERT tokenizers headed for ~50x total speedup as tokenizers v1 work continues — tomaarsen · 2026-10-15
[source] Tokenization refactor speeds up Sentence Transformers, biggest gains for static embedding models — tomaarsen · 2026-10-15
[source] Hugging Face tokenizers: another ~5x speedup for BERT tokenizers on top of v1 — tomaarsen · 2026-10-15

1 near-duplicate retellings: art_zucker