VisTW: a Traditional Chinese VLM benchmark for reading Taiwan — and an eval framework that caught a 36-point bug
piske_usagi · reddit · 2026-09-16
Mainstream VLM evals (MMBench, MMMU, POPE) are English or Simplified Chinese only, so models can ace them yet fail on Taiwanese road signs, receipts, health-insurance cards, or local textbook diagrams. VisTW (NTU MiuLab, arXiv 2503.10427, CC BY 4.0) fills the gap with two subsets: VisTW-MCQ, 4,770 Traditional Chinese multiple-choice questions from real exams across 21 subjects (charts, circuit diagrams, sheet music, medical imaging, maps), and VisTW-Dialogue, 141 open-ended questions about everyday Taiwanese scenes scored 0–10 by an LLM judge. On the official leaderboard, gemma-3-12b-it scores 0.4863 MCQ / 3.94 Dialogue; o3 tops at 0.7769 / 6.9878 (leaderboard stale since April 2025). The author's team added VisTW support to the open-source Twinkle Eval framework (v2.10.0) and validated against the official implementation: same model, endpoint, and 253 sampled questions, 81.01% vs 80.56% accuracy. Alignment surfaced a real bug: the official prompt's "答案: $字母" format made models echo the dollar sign, breaking their extractor, which fell back to a loose pattern and plucked letters from reasoning text — one subject scored 36 points low, and the bug suppressed scores across five vision benchmarks while a 21-question smoke set caught nothing. Also: they report unparsedrate instead of the official LLM-based fallback parser, and reasoning VLMs need headroom (19% truncated at maxtokens 2048, costing 19 points; templates ship at 8192).
More from Models
- Open-source VoxLock tops Krisp voice isolation benchmark with 4.95% WER — Arbaz Siddiqui · 2026-11-02
- Anthropic's 85% inference margin sparks debate on open source's price-pressuring role — code_star · 2026-11-02
- User reports xAI appears to have quietly raised weekly usage limits, dropping his meter from 93% to 47% — XFreeze · 2026-11-02
- Meituan's LongCat-2.5 lands as 1.6T MoE agent model, with DeepSeek copycat speculation — teortaxesTex · 2026-11-02
- Asked what AI firms want you to stop owning, Claude said: your own compute — alex_verem · 2026-11-02
- Is Hermes Still Worth It? Codex's $200 Tier Now Overlaps Most of Its Value — Foreign_Lead_3582 · 2026-11-02