HF tokenizers cuts special-token memory 1,000×, boosts decode speed 26%
art_zucker · x · 2026-11-03
New in Hugging Face tokenizers: special tokens now take 1,000× less memory. Previously each added token got its own slot — gpt-oss's 21 added tokens (ids 199,998–200,018) occupied 200,019 slots. Benchmarks across 6 popular Hub tokenizers (AMD EPYC 7R13) show -17% tokenizer memory, -19% load time, +26% decode speed. v1 rc ships soon.
More from Infra
- Satire: companies borrow a 'quadrillion' to build 420 data centers in 6 months — asymco · 2026-11-03
- SF billboards slam cloud AI for stealing data; blogger predicts decade of on-device iPhone assistants — brandon_xyzw · 2026-11-03
- DeepSeek Held #1 on OpenRouter 23 of 25 Weeks, Now 32% of Tokens — teortaxesTex · 2026-11-03
- Running DeepSeek-V4.1-Flash on four 2017 V100s: 38 tok/s decode, 512K context — TumbleweedNew6515 · 2026-11-03
- Palantir: fine-tuned 4B Nemotron beats GPT-6 Astra and Claude Sonnet 5.5 at 1/200th the cost — eliano · 2026-11-03
- AI memory crunch kills the $100 phone: sub-$100 shipments down 60% — demian_ai · 2026-11-03