Chartography: A New Benchmark for Pro Chart Understanding
echen · x · 2026-07-17
Existing chart benchmarks (like ChartQA) are nearing resolution on basic charts (80-90% accuracy) but fail to reflect the challenges of real-world professional scenarios.
The research team introduced Chartography, a benchmark designed specifically to test frontier models on understanding professional charts (e.g., oncologist survival curves, contour maps, candlestick charts, Sankey diagrams). Test results show that even the best-performing models score below 45%. The authors note that models often experience total computational collapse due to misreading grid lines, and excessive reasoning only prolongs these errors.
Related event: Chartography: A New Benchmark for Professional Chart Understanding(3 posts)→
More from Models
- giffmana: the env being used in training is part of the point — giffmana · 2026-09-11
- awesome-llm-leaderboards: an open-source directory of LLM leaderboards, pricing tables, comparison tools — Last_Establishment_1 · 2026-09-11
- Anthropic claims it works to keep eval environments unidentifiable to models — MaxKannen · 2026-09-11
- Nex N2.5 Pro released on Hugging Face with 407GB of weights — jinnyjuice · 2026-09-11
- RoMa v2 image matching model unveiled in the usual black poster — ducha_aiki · 2026-09-11
- OpenAI rated Astra 'Critical' for cyber capabilities — and admits it's harder to monitor — theguywhobuilds · 2026-09-11