ttok 1.0: experiment suggests GPT-6 shares the GPT-5 tokenizer, all models match on 31 fixtures
Simon Willison · rss · 2026-10-09
Simon Willison shipped ttok 1.0 after realizing version 0.4 still defaulted to the GPT-4 tokenizer instead of the GPT-5/GPT-6 family.
- OpenAI hasn't officially confirmed GPT-6 uses the GPT-5 tokenizer (there's an angry issue about it)
- William Liu's experiment offers evidence: all seven models—GPT 5.5, 5.6 Sol/Terra/Luna, and 6 Astra/Sol/Luna—report the same 44,794 tokens and match on every one of 31 fixtures
- Conclusion: GPT-6 introduces no input-count change on this corpus; the tokenizer is likely unchanged from the GPT-5 family
More from Models
- Boundless offers abliterated DeepSeek V4.1 Flash for defensive security testing — MoonL88537 · 2026-10-23
- Anthropic uses Claude to formally verify security properties in 100+ open-source libraries at RISC-V assembly level — AnthropicAI · 2026-10-23
- Are models in an uncanny valley? Researcher warns users to stay vigilant — danielrock · 2026-10-23
- Uncensored Swift 1.5 Qwen 3.8 Flash Next Released as 4-bit GGUF with Strata-Ready Artifacts — jjusko20 · 2026-10-23
- Rumor: Anthropic training a model matching OpenAI's internal one; big launches in December — imjustnewatai · 2026-10-23
- AI researcher: Claude finally finished tedious research tasks I gave up on years ago — MikePFrank · 2026-10-23