Comparing Anthropic vs OpenAI token counts is flawed: tokenizer efficiency differs
JoshPurtell · x · 2026-10-06
Continuing his thread on Codex subscription subsidies, Josh Purtell criticizes an analysis article for directly comparing token counts between Anthropic and OpenAI models — despite Anthropic admitting its tokenizer isn't especially "efficient."
His point: cross-provider cost analyses built on raw token counts are unreliable; any breakeven calculation between subscription and API pricing must first normalize the measurement basis.
More from Models
- Reflection AI's Beam praised as competitive without Claude distillation — teortaxesTex · 2026-10-06
- "The Models Are Still Phenomenally Dumb": Jagged Abilities Keep Fooling Us — generativist · 2026-10-06
- GLM 5.3 lands on Amazon Bedrock: 753B MoE with stronger coding and standout cybersecurity scores — AWS ML Blog · 2026-10-06
- CivBench pits LLMs against each other in Civilization V: GLM-5.3 beats Opus-5.5, open-weight Qwen-3.8-27B impresses — vox-deorum · 2026-10-06
- Reddit Thread Weighs Tools for Uncensoring Local AI Models, From Heretic to Abliterated Weights — BruceMilk · 2026-10-06
- Dev argues Codex subscriptions may be subsidized: here's how to compute the breakeven — JoshPurtell · 2026-10-06