Hugging Face Ships tokenizers v1, Often Tens of Times Faster Than v0.23
art_zucker · x · 2026-09-21
Hugging Face released the first major version of its tokenizers library, targeting workloads where tokenization starves models of data: massive training runs, concurrent serving, and repeated long-input processing. v1 is often tens of times faster than v0.23, with better multi-thread scaling, all-language support, and minimal package size and memory use. The team credits ideas from rival open-source tokenizers (tiktoken, kitoken, gigatoken, etc.) and thanks NVIDIA, IBM, and the ExecuTorch team for patches and cross-hardware testing. Full benchmarks accompany the release.
Related event: Hugging Face Ships tokenizers v1 RC with Up to 30x Speedup(8 posts)→
More from Infra
- Photoroom reclaims ~1,000 idle GPU hours a week with SkyPilot across Slurm and Kubernetes — skypilot_org · 2026-11-12
- Cloudflare Talk: Swapping Some Agent LLM Calls for Fast Models Like Jev Gives Big Speedups — michellechen · 2026-11-12
- Cloudflare Trains Its Own Clef and Clairic Models to Complete Its Agent Stack — michellechen · 2026-11-12
- AWS open-sources 23 production-ready GenAI assets, cutting build time from months to weeks — AWS ML Blog · 2026-11-12
- Sixth Street CEO: ~$200B of Chips Sitting in Warehouses as Data Centers Lag Behind — SumitGup · 2026-11-12
- AWS CEO Matt Garman: Space Data Centers Are 10-20 Years Away, Likely Not Economically Feasible — rohanpaul_ai · 2026-11-12