Hugging Face ships tokenizers v1, often tens of times faster than v0.23
ariG23498 · x · 2026-09-21
Hugging Face released tokenizers v1, a performance-focused rewrite: encoding, decoding and multi-thread scaling are often tens of times faster than v0.23, with multi-language support and minimal package size. Ideas drawn from the open-source ecosystem (tiktoken, kitoken, gigatoken etc.), with patches from IBM, NVIDIA and the ExecuTorch team. Includes measured benchmarks.
Related event: Hugging Face Ships tokenizers v1, Up to 30x Faster(8 posts)→
More from Infra
- SGLang's hicache: use an L3 storage cache to keep KV cache alive across local model swaps — TheZachMueller · 2026-09-22
- Egypt's AI Ecosystem Hits Production Scale With $400M Data Center, 10x NVIDIA Learner Growth — nordicinst · 2026-09-22
- RTX Pro 6000 vs a used 3090 vs cloud rental: the LoRA training math — big-in-jap · 2026-09-21
- ComfyUI GPU rental showdown: Modal's 35s cold starts and free 1TiB beat RunPod — ronalder100 · 2026-09-21
- Meta partners with Arm on Arm AGI CPU, its first AI-era data center CPU — bookwormengr · 2026-09-21
- Starlink is becoming core infrastructure for rural education across Latin America — XFreeze · 2026-09-21