Reverse-Engineering Claude's Tokenizer: Smaller Vocab but Higher Inference Cost
kalomaze · x · 2026-08-14
A developer has successfully reverse-engineered and exactly reproduced Anthropic's Claude tokenizer (ctok). The tool accurately matches token counts across 500+ natural languages and 22 programming languages entirely offline.
Key Findings:
- Vocabulary Shrinkage: Claude v3 had an estimated vocabulary of 49,152 entries, but the newer v4.7 (Claude 5) drops to just 15,000 entries (including image tokens).
- The Cost Trade-off: While Claude 5's vocabulary is about 12× smaller than OpenAI's o200k, its token bill is 1.24× to 2.44× larger on identical samples.
- Pricing Impact: Since Claude Opus 5 and GPT-5.6 Sol both charge $5 per million input tokens, the larger token ratio directly translates to higher API costs for Claude users.
More from Models
- Devs say LLM coding mistakes are a different class: overcomplication, inferred limits, worse at higher reasoning — Mu5_ · 2026-10-16
- ChatGPT UI screenshot reveals Wallet coming soon plus Email, Calendar, Health and Finances integrations — jxnlco · 2026-10-16
- Why the next OpenAI and Anthropic models may still disappoint: no native audio or video input — imjustnewatai · 2026-10-16
- Thousands Hit by ChatGPT Standard Voice Mode Regression: Wrong Voice, Interruptions — shaunralston · 2026-10-16
- Gemini Argon 4 Quota Exhausted, Reset Countdown Shows Nearly 7 Days — ssh4net · 2026-10-16
- AINews: Meta ships 816B Muse Spark Open; Goodfire argues neural nets are already clean — Latent Space · 2026-10-16