Tokenmaxxing BERT: how skipping ASCII normalization made tokenization 50x faster
tomaarsen · x · 2026-10-15
Hugging Face is reviving its tokenizers library toward v1.0.0: classic BPE tokenizers now encode up to 30x faster than v0.23, but BERT's WordPiece path lagged at 8.6x. Simon Brandeis's new post explains how BERT tokenization got another 5.7x faster — about 50x total (still in PR stage). Benchmarks show bert-base-uncased English encoding jumping from 2.6 to 132 MB/s (50.8x).
Key points:
- BERT-like encoders power embeddings, search/RAG, rerankers and classifiers; the three most-downloaded Hub models all run BERT's normalizer (all-MiniLM-L6-v2 at 218M monthly downloads, ms-marco-MiniLM-L6-v2 at 79M, bge-small-en-v1.5 at 59M).
- Across the top 10,000 Hub models, BPE tokenizers account for 842M monthly downloads and WordPiece 627M — all passing through BERT's normalizer, making normalization the key bottleneck.
- The post details how the tokenizers work, where the implementation was bottlenecked, and includes a browser demo to measure speed before/after the refactor.
Related event: Hugging Face Turbocharges BERT Tokenizer With Up to 50x Speedups(4 posts)→
More from Infra
- PS5LM runs Qwen 3.8 on a jailbroken PS5 at ~20 tok/s via llama.cpp and Vulkan — lxfater · 2026-10-15
- Cerebras CEO: OpenAI went with Nvidia because Cerebras lacked capacity for GPT-6 — firstadopter · 2026-10-15
- patio11 on multi-agent dev: clearly the future, and we'll buy so much inference — IgorCarron · 2026-10-15
- Hugging Face Hub makes model tensor inspection up to 20x faster, DeepSeek-V3.2 drops from 9.2s to 0.45s — victormustar · 2026-10-15
- RTX 5060 Ti 16GB surges up to 144% in Japan, hitting $1,292 as RTX 5080 tops $2,400 — chemist_slime · 2026-10-15
- Fish Audio Drama 3 lands in vLLM-Omni: 128 concurrent streams on one H200 — vllm_project · 2026-10-15