Actual Releases Assembly-Based Tokenizer toks, Up to 151x Faster Than Hugging Face

Actual Computer has released toks, a production tokenizer largely written in pure assembly that returns token ids identical to Hugging Face tokenizers and tiktoken while running 13-151x faster, achieving up to 1.8 GB/s encoding on 8 GB10 cores.

2026-10-07 ~ 2026-10-07 · 3 related posts

• Actual ships toks, an asm-powered tokenizer claiming 151x speedup over Hugging Face — markjeffrey · 2026-10-07
• toks tokenizer claims 13x-151x speedup over Hugging Face, beats tiktoken everywhere — markjeffrey · 2026-10-07
• toks runs its tokenizer in pure assembly: 1.8 GB/s on 8 Zen 5 cores — cephaloform · 2026-10-07