2026-09-25
FinChart-Bench has 1,200 real financial charts and 7,016 single-token items; Claude Sonnet 4 leads at 84.32% average but only 63.59% on numeric QA.
Chart-understanding boards already exist: FigureQA, ChartQA, ChartX, mostly open-domain or scientific plots. Financial charts are almost never tested on their own. Bars, lines, and pies in earnings slides carry temporal structure and jargon, and the number often lives on the axis rather than next to the mark. That is a different difficulty from generic chart QA.
Two engineering problems sit on top. Answer formats are fuzzy ("140 Million" versus "140M" versus "$140M"), so GPT-as-judge becomes part of the metric. Many boards also let models write the questions and score the answers. Finance wants exact numbers; fuzzy answers do not clear a trading-floor bar.
FinChart-Bench pulls charts from 2015–2024 public-company call slides, including Bloomberg and official sites. Qwen2.5-VL-7B boxes charts, yielding about 10k images a year and more than 130k in total. Humans then filter on four hard rules: it must be a chart, complete and legible, high-resolution, and neither trivial nor unreadable. About 7k images survive per year; 120 of the cleanest are kept per year, 1,200 in total. Numbered bars are about 40%, bar-plus-line about 26%, across 14 types.
GPT-4.1 writes six items per chart: two true/false, two multiple choice, two numeric QA. Answers are forced to a single token: True/False, A/B/C/D, or one number. Reasoning traces are stored off to the side and ignored by the scorer. A second human pass fixes wrong answers and drops bad questions, 7,200 pairs down to 7,016 (2,383 TF, 2,349 MC, 2,284 QA). The authors annotated themselves: about 1,000 images an hour for filtering, about 80 pairs an hour for QA, March to June 2025.
Twenty-six models are scored: 13 general open-source, five chart-finetuned, plus closed APIs. The output format is Result = [[ answer ]]. The metric is exact match, then a question-count-weighted average.
On true/false and multiple choice, strong open-source models already touch closed APIs. Mistral 3.1 (24B) hits 84.17% MC, above GPT-4o's 77.57%. Qwen2.5-VL-7B averages 72.16%, close to GPT-4o's 76.62%. QA is the ceiling for everyone: best open-source is Llama 4 at 59.78%, best closed is Claude Sonnet 4 at 63.59%. Average leader is Claude Sonnet 4 at 84.32%.
Family upgrades are not monotone. Llama 4's average (69.89%) beats Llama 3.2 (44.65%), but MC falls from 64.00% to 59.70%. GPT-4.1 mini scores 86.77% MC, above the more expensive GPT-4.1 at 83.19%. Chart-finetuned models are close to unusable: UniChart 17.52% average, ChartGemma 16.87%, most MC scores under 10%, mainly because they ignore the output format.
Numbered bars and lines are easy; axis-aligned line, area, and scatter plots collapse on QA. Qwen2.5-VL's extraction stage produced over 10k charts a year; humans threw away nearly half. The models are not ready to be automatic judges.
| Model | TF / MC / QA | Average |
| Claude Sonnet 4 | 96.94 / 91.66 / 63.59 | 84.32 |
| GPT-5 | 97.44 / 92.13 / 61.37 | 83.92 |
| Qwen2.5-VL 7B | 95.09 / 82.81 / 37.29 | 72.16 |
| Llama 4 17B | 89.56 / 59.70 / 59.78 | 69.89 |
| UniChart | 51.05 / 0.0 / 0.57 | 17.52 |
This is the first chart board aimed at real financial figures and fully human-checked twice. Single-token exact match makes the ranking reproducible without a GPT judge. For anyone shipping earnings QA, the takeaway is concrete: closed models already sit at 97% on true/false; the gap is reading axes and computing, not reading legends. Chart-specific finetunes that burn instruction following can lose to a generic 7B.
It is a domain benchmark, not a methods paper. Twelve hundred images buy trustworthy labels, not a new architecture.
Scale is capped by human labour: 71k clean charts were cut to 1,200. Questions start as GPT-4.1 drafts, so the phrasing may keep a generator accent. Single-token scoring ignores reasoning quality and only checks the final number. The slides are English corporate decks, not Chinese filings, research-note screenshots, or interactive terminals. Chart-finetuned models fail on format; the paper blames aggressive finetuning for killing instruction following, but those models were not retested with a gentler prompt, so that remains a hypothesis. The spatial-reasoning claim is a slice of accuracy by chart type, not a controlled "read the axis" experiment.