Hugging Face ships tokenizers v1: 3-30x faster encoding and decoding
pcuenq · x · 2026-09-21
Hugging Face released tokenizers v1 (Rust), delivering 3-30x faster encode/decode versus v0.23, with an opt-out training crate, new algorithms, and extensive benchmarks; it will land in transformers soon.
- Motivation: as models get faster and workloads scale, tokenization can starve GPUs of data — v1 aims to make it never the bottleneck;
- Biggest wins: massive-dataset training, concurrent serving, repeated long-input processing;
- The team credits ideas from open-source projects like tiktoken, gigatoken, and kitoken;
- IBM, NVIDIA, and the ExecuTorch team contributed patches.
Related event: Hugging Face Ships tokenizers v1 RC with Up to 30x Speedup(8 posts)→
More from Infra
- Cloudflare Talk: Swapping Some Agent LLM Calls for Fast Models Like Jev Gives Big Speedups — michellechen · 2026-11-12
- Cloudflare Trains Its Own Clef and Clairic Models to Complete Its Agent Stack — michellechen · 2026-11-12
- AWS open-sources 23 production-ready GenAI assets, cutting build time from months to weeks — AWS ML Blog · 2026-11-12
- Sixth Street CEO: ~$200B of Chips Sitting in Warehouses as Data Centers Lag Behind — SumitGup · 2026-11-12
- AWS CEO Matt Garman: Space Data Centers Are 10-20 Years Away, Likely Not Economically Feasible — rohanpaul_ai · 2026-11-12
- CoreWeave Database Goes GA: Managed PostgreSQL Running Inside Your CKS Cluster Next to GPUs — _ScottCondron · 2026-11-12