Hugging Face ships tokenizers v1, often tens of times faster than v0.23
ariG23498 · x · 2026-09-21
Hugging Face released tokenizers v1, a performance-focused rewrite: encoding, decoding and multi-thread scaling are often tens of times faster than v0.23, with multi-language support and minimal package size. Ideas drawn from the open-source ecosystem (tiktoken, kitoken, gigatoken etc.), with patches from IBM, NVIDIA and the ExecuTorch team. Includes measured benchmarks.
Related event: Hugging Face Ships tokenizers v1 RC with Up to 30x Speedup(8 posts)→
More from Infra
- Photoroom reclaims ~1,000 idle GPU hours a week with SkyPilot across Slurm and Kubernetes — skypilot_org · 2026-11-12
- Cloudflare Talk: Swapping Some Agent LLM Calls for Fast Models Like Jev Gives Big Speedups — michellechen · 2026-11-12
- Cloudflare Trains Its Own Clef and Clairic Models to Complete Its Agent Stack — michellechen · 2026-11-12
- AWS open-sources 23 production-ready GenAI assets, cutting build time from months to weeks — AWS ML Blog · 2026-11-12
- Sixth Street CEO: ~$200B of Chips Sitting in Warehouses as Data Centers Lag Behind — SumitGup · 2026-11-12
- AWS CEO Matt Garman: Space Data Centers Are 10-20 Years Away, Likely Not Economically Feasible — rohanpaul_ai · 2026-11-12