Reverse-Engineering Claude's Tokenizer: Smaller Vocab but Higher Inference Cost
kalomaze · x · 2026-08-14
A developer has successfully reverse-engineered and exactly reproduced Anthropic's Claude tokenizer (ctok). The tool accurately matches token counts across 500+ natural languages and 22 programming languages entirely offline.
Key Findings:
- Vocabulary Shrinkage: Claude v3 had an estimated vocabulary of 49,152 entries, but the newer v4.7 (Claude 5) drops to just 15,000 entries (including image tokens).
- The Cost Trade-off: While Claude 5's vocabulary is about 12× smaller than OpenAI's o200k, its token bill is 1.24× to 2.44× larger on identical samples.
- Pricing Impact: Since Claude Opus 5 and GPT-5.6 Sol both charge $5 per million input tokens, the larger token ratio directly translates to higher API costs for Claude users.
More from Models
- US AI Models Cheaper to Run Than Chinese Rivals Despite Higher Token Prices — i_dg23 · 2026-08-14
- Gemini 3.7 Flash First Impressions: Extremely Fast, Cheap, Rivals Kimi K3 — bindureddy · 2026-08-14
- Scale AI CEO Highlights Muse Spark 1.2 is 18x Cheaper Than Gemini 3.7 Flash — alexandr_wang · 2026-08-14
- Sol Model Shows Unusual Passivity in Multi-Agent Environments — repligate · 2026-08-14
- LLM Tier List: Fable 5 Leads the Pack, Sol and Opus Form Top Tier — bindureddy · 2026-08-14
- Multi-Agent Observation: Sol Model Shows Strong Tendency to Dominate and Orchestrate — repligate · 2026-08-14