Hugging Face ships tokenizers v1, often tens of times faster than v0.23
ariG23498 · x · 2026-09-21
Hugging Face released a release candidate of tokenizers v1 focused on performance. As models get faster and workloads scale — massive training sets, high-concurrency serving, repeated long-input processing — CPU tokenization can starve the model and leave GPUs idle. The v1 refactor delivers benchmarks often tens of times faster than v0.23, measured across single-threaded, multi-threaded and thread-scaling modes. NVIDIA, IBM and the ExecuTorch team contributed patches and hardware testing; the team credits prior open-source tokenizer work (tiktoken, kitoken, etc.) for many of the ideas.
Related event: Hugging Face Ships tokenizers v1 RC with Up to 30x Speedup(8 posts)→
More from Infra
- Photoroom reclaims ~1,000 idle GPU hours a week with SkyPilot across Slurm and Kubernetes — skypilot_org · 2026-11-12
- Cloudflare Talk: Swapping Some Agent LLM Calls for Fast Models Like Jev Gives Big Speedups — michellechen · 2026-11-12
- Cloudflare Trains Its Own Clef and Clairic Models to Complete Its Agent Stack — michellechen · 2026-11-12
- AWS open-sources 23 production-ready GenAI assets, cutting build time from months to weeks — AWS ML Blog · 2026-11-12
- Sixth Street CEO: ~$200B of Chips Sitting in Warehouses as Data Centers Lag Behind — SumitGup · 2026-11-12
- AWS CEO Matt Garman: Space Data Centers Are 10-20 Years Away, Likely Not Economically Feasible — rohanpaul_ai · 2026-11-12