Hugging Face engineers make BERT tokenization 50x faster by skipping ASCII
art_zucker · x · 2026-10-15
Hugging Face engineer Simon Brandeis details how the team made BERT-like model tokenization up to 5.7x faster on top of the tokenizers library overhaul — roughly 50x faster than v0.23 in total (changes still in PR stage).
Context
- The team has been reviving the 🤗 tokenizers library targeting higher encoding throughput and a smaller footprint, heading toward a v1.0.0 release.
- BPE tokenizers (used by most LLMs) are already up to 30x faster than v0.23; WordPiece (BERT-style) only got 8.6x, which the author set out to fix.
Why BERT matters
BERT-like encoders power embeddings, search, RAG, rerankers, and classifiers. The three most-downloaded models on the Hub (all-MiniLM-L6-v2 at 218M monthly downloads, ms-marco-MiniLM-L6-v2 at 79M, bge-small-en-v1.5 at 59M) all use BERT's normalizer; across the top 10,000 models, WordPiece accounts for 627M monthly downloads.
Key technique
As the title hints, skipping ASCII-related processing in BERT's normalizer was the key to the extra 5.7x speedup, alongside other engineering optimizations.
Related event: Hugging Face's tokenizers v1 push makes BERT tokenizer ~50x faster(6 posts)→
More from Infra
- Cloudflare data: after strikes on Yandex data centers, malicious bots simply moved networks — banteg · 2026-10-15
- OpenAI is hiring for its UK infrastructure team, offering end-to-end ownership — whoiskatrin · 2026-10-15
- XERJ indexes code, docs and PDFs in one command, saving agents 40x tokens vs grep — xerj-org · 2026-10-15
- TokenSpeed, a 'speed-of-light' LLM inference engine for Blackwell and Rubin, hits 2.3k stars — lightseekorg · 2026-10-15
- Egress surprise: moving a petabyte to the cloud is free, pulling it back costs tens of thousands — DavidLinthicum · 2026-10-15
- BERT tokenizer gets 50x speedup by skipping text normalization, not optimizing it — LysandreJik · 2026-10-15