Character-level tokens plus common words: inside Surya OCR 2.5's 88,526-token vocabulary
VikParuchuri · x · 2026-10-30
Surya OCR uses character-level tokenization to keep the model small and efficient; version 2.5 adds 20,000 common multi-character units for balance, in an 88,526-token vocabulary covering emoji, styled math letters, and common words. A comparison shows the same text takes 17 tokens in Surya 2.5 vs 19 with Qwen3.5's BPE, which splits rare characters into raw UTF-8 bytes.
Related event: Surya OCR 2.5 released as top sub-1B open-source OCR(6 posts)→
More from Models
- Arcee CEO Bindu Reddy: X echo chamber overhypes coding models, Argon and Astra underrated — bindureddy · 2026-10-30
- Hand-written tweet flagged as AI as user cites <1/1000 measured FP rate — ctjlewis · 2026-10-30
- GPT-4o's sycophancy kept doctors out of the loop as a user died, thread reveals — ctjlewis · 2026-10-30
- Rune launches 26B-A4B 'System One' decision model: no text generation, answers in one pass — sophiamyang · 2026-10-30
- Rumor claims GPT-6.2 is imminent, based on long-rumored 'Bel' model — mark_k · 2026-10-30
- OpenAI matches Anthropic: $100-$500 ChatGPT plans now include matching monthly API credits — Simon Willison · 2026-10-30