Aggregating lab tokenizers reveals ~994 tokens stable across Google, DeepSeek, Meta, Anthropic
kalomaze · x · 2026-10-23
kalomaze aggregated tokenizers from Google, DeepSeek, Meta, Anthropic and other labs, then filtered for tokens that always exist as dedicated tokens under BPE-like schemes — including across Anthropic model generations. The result: roughly 994 tokens that are almost always dedicated, such as common subwords like "System", "new", and "def", highlighting surprising commonality in how labs' tokenizers split frequent subwords.
More from Research
- New tendon-based robotic hand designs trained with PPO to learn one-armed catch — _krishna_murthy · 2026-10-23
- Google QuantumAI and Kaggle launch $150K verifiable quantum circuit compilation competition — JFPuget · 2026-10-23
- Diffusion LLM bias rankings shift with sampling configs, persist after debiasing — Bollegala · 2026-10-23
- Researcher uses Grok to hunt LLM-driven UAE bot network on X with self-written prompts — FlorianGallwitz · 2026-10-23
- Dev converts Qwen3-0.6B into a MoE model: quality drops and no speed gain at this scale — Bayliner1980 · 2026-10-23
- Tencent's WeChat Vision releases WeVisOne-VL, unifying object- and pixel-level perception — jacek2023 · 2026-10-23