Hugging Face Ships tokenizers v1, Often Tens of Times Faster Than v0.23
art_zucker · x · 2026-09-21
Hugging Face released the first major version of its tokenizers library, targeting workloads where tokenization starves models of data: massive training runs, concurrent serving, and repeated long-input processing. v1 is often tens of times faster than v0.23, with better multi-thread scaling, all-language support, and minimal package size and memory use. The team credits ideas from rival open-source tokenizers (tiktoken, kitoken, gigatoken, etc.) and thanks NVIDIA, IBM, and the ExecuTorch team for patches and cross-hardware testing. Full benchmarks accompany the release.
Related event: Hugging Face Releases Tokenizers v1 with Major Speedups(3 posts)→
More from Infra
- AI intern cuts job boot time ~60%, saving ~5 CPU-days per day — DanielLockyer · 2026-09-21
- Tim Dettmers' lab announces open-source week: 2 frameworks, 4 papers for frontier AI on local hardware — Tim_Dettmers · 2026-09-21
- I was wrong: you can run LLMs on this device and even do development — gnukeith · 2026-09-21
- Backblaze CEO on CoreWeave deal: 'a land grab in the AI ecosystem' — TiernanRayTech · 2026-09-21
- Training engineer tip: save checkpoints to K8s PVC — and don't alert on that PVC — cto_junior · 2026-09-21
- Baseten CEO: agents burning tokens will make inference 'core IP' for every company — baseten · 2026-09-21