VisTW: a Traditional Chinese VLM benchmark for reading Taiwan — and an eval framework that caught a 36-point bug
piske_usagi · reddit · 2026-09-16
Mainstream VLM evals (MMBench, MMMU, POPE) are English or Simplified Chinese only, so models can ace them yet fail on Taiwanese road signs, receipts, health-insurance cards, or local textbook diagrams. VisTW (NTU MiuLab, arXiv 2503.10427, CC BY 4.0) fills the gap with two subsets: VisTW-MCQ, 4,770 Traditional Chinese multiple-choice questions from real exams across 21 subjects (charts, circuit diagrams, sheet music, medical imaging, maps), and VisTW-Dialogue, 141 open-ended questions about everyday Taiwanese scenes scored 0–10 by an LLM judge. On the official leaderboard, gemma-3-12b-it scores 0.4863 MCQ / 3.94 Dialogue; o3 tops at 0.7769 / 6.9878 (leaderboard stale since April 2025). The author's team added VisTW support to the open-source Twinkle Eval framework (v2.10.0) and validated against the official implementation: same model, endpoint, and 253 sampled questions, 81.01% vs 80.56% accuracy. Alignment surfaced a real bug: the official prompt's "答案: $字母" format made models echo the dollar sign, breaking their extractor, which fell back to a loose pattern and plucked letters from reasoning text — one subject scored 36 points low, and the bug suppressed scores across five vision benchmarks while a 21-question smoke set caught nothing. Also: they report unparsedrate instead of the official LLM-based fallback parser, and reasoning VLMs need headroom (19% truncated at maxtokens 2048, costing 19 points; templates ship at 8192).
More from Models
- OpenAI's roon predicts everyone will have Astra-level capabilities within a month or two — Tolopono · 2026-09-16
- As LMs mature, the work reduces to two things: data and infra — saurabh_shah2 · 2026-09-16
- Emergent behavior: telling Astra to be fast makes it fast without cutting corners — reach_vb · 2026-09-16
- Frontier coding agents now write code no human can read, warns tszzl — tszzl · 2026-09-16
- ATLAS Finance benchmark: best of 11 frontier models scores 12% vs humans' 100% — garrytan · 2026-09-16
- Google engineers can now use Claude internally alongside Gemini — VraserX · 2026-09-16