Hugging Face tokenizers: another ~5x speedup for BERT tokenizers on top of v1
tomaarsen · x · 2026-10-15
After the tokenizers v1 rewrite brought major speedups, Simon Brandeis is working on another 5x improvement for BERT tokenizers — roughly 50x faster than v0.23 in total. tomaarsen highlights the accompanying blog post explaining how these tokenizers work, where the implementation was bottlenecked, and how it improved, plus an in-browser demo to measure tokenization speed before and after the refactor.
Related event: Hugging Face Turbocharges BERT Tokenizer With Up to 50x Speedups(4 posts)→
More from Models
- Fireworks AI CEO: 0.1% per-turn numerical drift compounds to ~10% over 100 agent turns — AccBalanced · 2026-10-15
- kalomaze argues ensembled synthetic data can cover data-scarce domains — kalomaze · 2026-10-15
- Newer image models are easier to jailbreak, with 2x more attacks getting through — Scobleizer · 2026-10-15
- Gemini controls a real robot from one prompt: SSH in, find joints, learn from demo — heiga_zen · 2026-10-15
- Tokenization refactor speeds up Sentence Transformers, biggest gains for static embedding models — tomaarsen · 2026-10-15
- Open-weights TYGER model combines ASR with full-duplex turn control, 250ms latency — KokaOP · 2026-10-15