MathVis benchmark: 512 research-level math figures, best model scores just 31%
青稞AI · wechat · 2026-10-15
SoTALab released MathVis (arXiv:2610.12954), a benchmark testing whether LLMs can turn advanced math statements into precise figures: given only a math description, models must emit compilable TikZ or renderable SVG, with no web or plotting tools.
- Composition: 512 tasks across 27 subfields (differential geometry 55, Euclidean geometry 52, Lie groups/representation theory 47, combinatorics 39), curated from 8,114 candidates via five filtering rounds with dual expert review. Scoring runs three gates: technical (compiles/renders), hard mathematical constraints (objects, relations, labels, global consistency) and soft readability/layout.
- Results (16 model configs, 3 generations each): Pass@1 leaders — GPT-6-Astra 31.25%, Claude Fable 5 26.56%, Gemini 3.1 Pro preview 22.85%. Strict delivery ranges 9.18%–30.73% while technical pass tops at 94.40% — most figures render, but fewer than a third are mathematically correct.
- Failure patterns: on a Rouché's theorem task even GPT-5.6-sol drew the circle wrong; models lean on symbolic reasoning and rarely use the figure to reason; 'all elements present but disorganized'; errors concentrate in mathematical relations (missing/reversed arrows, misbound labels). Agent environments brought no consistent gains.
- Motivation: 'compiles' is an insufficient training target — atomic scoring enables hard negatives and verifier-guided fine-tuning, pushing models to treat figures as part of reasoning, not final illustrations.
More from Models
- ChatGPT web can actually listen to audio, but the same model can't via API or Codex — Possible_Statement84 · 2026-10-23
- MiniMax M3.1 Flash free for 3 days on 0G, with 1M context and 512K output — MiniMax_AI · 2026-10-23
- TLA+ expert corrects Grok: TLC only exhausts finite state spaces, not all behaviors — tianyin_xu · 2026-10-23
- Open LLM cost calculator shows Arabic tokenization and 6x price gaps across models — Moham-saed · 2026-10-23
- OpenAI blocks manual reset with 13% usage left: 'no reset needed right now' — AccomplishedSugar490 · 2026-10-23
- Meta's 816B Muse Spark Open scores 39, losing to 180B Qwen3.8-Flash-Next at 40 — BTA_Labs · 2026-10-23