Plotly and Mila launch DataAnalyticsBench: 43 tasks, 28 models on a 20M-row warehouse
Mila_Quebec · x · 2026-10-17
Plotly, with Mila researchers, launched DataAnalyticsBench—a benchmark purpose-built for AI data analytics, avoiding benchmaxxed SWE benchmarks and well-worn Iris/Titanic datasets.
Design: a 43-question exam where an AI agent answers against a 19.9M-row warehouse (44 tables, 554 columns) full of trick questions, messy data, and huge schemas, run in the Plotly Studio Embedded harness. 28 models tested.
Key results (as of Oct 15):
- GPT-6 Astra was first to ace it; GPT-6.1 Sol aced it 3 weeks later at 10% of the cost; GPT-6 Luna got a B+ at 5% of that
- Claude Haiku 5.5 scored 37/43 at $0.38/run; Claude Opus 5.5 and Meta's Muse Wonder (Oct 12) tied at 41/43; Gemini 4 Argon (Oct 13) also 41/43
- Open weights: Meta's Muse Spark Open hit 38/43; NVIDIA's Nemotron 3.5 Super VL scored 30/43 while becoming the fastest model
Sol and Luna currently hold the cost-accuracy frontier for closed-weight models.
More from Models
- Grok Bots Roll Out on Grok Web and Mobile, Filling the Gap OpenAI's Dots Leaves Open — testingcatalog · 2026-10-17
- Meta terms reportedly grant itself rights to scrape the open web for AI training — VeryWellVersed · 2026-10-17
- Codex users lose remote usage tracking, leaving only an edit button — No-Butterscotch-218 · 2026-10-17
- Dev asks why non-tech users prefer ChatGPT when Claude is 'far better' — prasenx · 2026-10-17
- ChatGPT's annoying regressions: garbled input, broken inline images, constant timeouts — ChristianKl · 2026-10-17
- Model tweaks barely move MMLU but easily break generative tasks — aryaman2020 · 2026-10-17