Hugging Face makes tokenizers 15.9x faster without changing a single token ID
jasonkneen · x · 2026-09-21
Hugging Face shipped a Rust rewrite of its tokenizers library delivering a 15.9x speedup while keeping every token ID identical. The gains come from a SIMD-based splitter replacing the regex engine, caching for repeated words, an allocation-free merge loop, and batched model calls — all behind the same API.
Related event: Hugging Face Ships tokenizers v1 RC with Up to 30x Speedup(8 posts)→
More from Infra
- Cloudflare Talk: Swapping Some Agent LLM Calls for Fast Models Like Jev Gives Big Speedups — michellechen · 2026-11-12
- Cloudflare Trains Its Own Clef and Clairic Models to Complete Its Agent Stack — michellechen · 2026-11-12
- Sixth Street CEO: ~$200B of Chips Sitting in Warehouses as Data Centers Lag Behind — SumitGup · 2026-11-12
- AWS CEO Matt Garman: Space Data Centers Are 10-20 Years Away, Likely Not Economically Feasible — rohanpaul_ai · 2026-11-12
- CoreWeave Database Goes GA: Managed PostgreSQL Running Inside Your CKS Cluster Next to GPUs — _ScottCondron · 2026-11-12
- micro1 scaled eval infra to 12× monthly tokens in 5 months, agent rollouts now start in 12 seconds — Exp_Mark · 2026-11-12