Actual ships toks, an asm-powered tokenizer claiming 151x speedup over Hugging Face
markjeffrey · x · 2026-10-07
- Actual Computer released its first public software, toks, a production-grade tokenizer "designed for the quadrillion-token era" that accepts any Hugging Face tokenizer config and returns ids identical to HF tokenizers or tiktoken.
- Speed comes from hand-written assembly hot paths with C glue and Python bindings: 13x–151x faster than Hugging Face tokenizers on a single core across fresh text, faster than tiktoken in all 195 comparable cells; Llama 3 English prose hits 269.8 MB/s on one GB10 core vs tiktoken's 30.3; Kimi K3 code reaches 474.3 MB/s, 124x tiktoken; multi-core tokspar encodes a 16 MiB input at 1.6 GB/s.
- The team argues "inhuman code" will become the default and plans to write more of toks in explicitly optimized assembly over time.
More from coding & agent
- Hermes Index scores: Opus 5.5 at $4.99/task leads, GPT 6 Astra costs $11.61 — NousResearch · 2026-10-07
- Hermes Index averages four suites including new in-house Hermes Bench — NousResearch · 2026-10-07
- Hermes Index methodology: same harness, reasoning set high where offered — NousResearch · 2026-10-07
- Nous Research launches Hermes Index agent leaderboard, Claude Opus 5.5 tops at 63.31 — NousResearch · 2026-10-07
- Coinbase Ships Agent Trading Features: Templates, TWAP Orders, and MCP Access — kleffew94 · 2026-10-07
- Dreadnode releases ScopeBench, a benchmark for agent scope adherence in offensive security — dyn___ · 2026-10-07