HF tokenizers v1.0.0-rc.4 restores Python APIs and encodes up to 17x faster
art_zucker · x · 2026-11-11
Hugging Face released tokenizers v1.0.0-rc.4, focused on Python.
- Ports back the 0.23 APIs that transformers and vLLM rely on: vocab access, special tokens, streaming decode
- Encoding up to 17x faster
- Breaking change from rc.3: the Tokenizer class no longer encodes directly — call tokenizer.compile() to build a frozen CompiledTokenizer; Tokenizer now holds an editable tokenizer.json config
- Feature parity with 0.23 on encoding/decoding is the last milestone before a stable 1.0.0
- Install: pip install --pre tokenizers==1.0.0rc4
More from Infra
- MacPaw's Elix claims fastest LLM inference on Apple silicon, winning all 33 benchmarks — JosephJacks_ · 2026-11-11
- Which tasks have you fully moved from cloud models to local ones? — FulcraDynamics · 2026-11-11
- X's recommender is multimodal: CLIP and MultimodalPostEmbedderV5 embeddings feed Phoenix retrieval — cocktailpeanut · 2026-11-11
- NEAR AI discloses breach: SGLang RCE flaw let attacker reach infra, ~100 users' data exposed — bingxu_ · 2026-11-11
- Bittensor launches Gamma Tokens for native agent payments across subnets — markjeffrey · 2026-11-11
- Meta's Muse Spark Open, 816B params with 1M context, goes live on ai& inference — DavidBennett__ · 2026-11-11