FT Report Claims AI Chatbots Mostly Wrong on Finance; Mollick Questions Methodology

An FT-cited study by Saturn claimed that leading AI models including ChatGPT, Claude, Copilot, Grok and Gemini give wrong answers to finance questions 57% of the time on average, and 88% on complex multi-step calculations. Ethan Mollick, after reviewing the report, questioned its methodology and conflicts of interest, turning the story into a broader debate over the credibility of AI evaluations.

Confirmed

Core figures: 57% average error rate on finance questions and 88% on complex multi-step problems, relayed by Gary Marcus, John Crickett and gerardsans, sourced from the FT report of Saturn's research.
Mollick noted the report came from a company selling a competing AI financial product, disclosed limited information with few sample questions, and contradicted recent research showing AI performs well on financial advice.
Mollick called for governments to directly evaluate models and publish results rather than leaving judgment to interested parties.

Unconfirmed

Given the methodological and conflict-of-interest concerns, the reliability of the 57% and 88% figures themselves remains unconfirmed.

Why it matters

The episode highlights both the reliability risks of AI models in professional finance and the flaws of an evaluation ecosystem dominated by vendors or competitors.
Commenters including KhoaVuUmn (sarcastically) and gerardsans used the report to criticize AI hype and over-trust, stressing caution when using AI in professional domains.

2026-09-20 ~ 2026-09-21 · 6 related posts

Primary sources

Mollick challenges FT claim that AI chatbots fail most financial queries — emollick · 2026-09-20 03:35 (UTC+08:00)
Ethan Mollick: governments should directly evaluate AI models instead of trusting third parties — emollick · 2026-09-20 03:42 (UTC+08:00)
Study: Top AI Models Get Financial Questions Wrong 57% of the Time — GaryMarcus · 2026-09-20 08:24 (UTC+08:00)

[source] Mollick challenges FT claim that AI chatbots fail most financial queries — emollick · 2026-09-20
[source] Ethan Mollick: governments should directly evaluate AI models instead of trusting third parties — emollick · 2026-09-20
FT: AI chatbots give wrong answers to financial queries 'most of the time' — SatelliteNetSec · 2026-09-20
[source] Study: Top AI Models Get Financial Questions Wrong 57% of the Time — GaryMarcus · 2026-09-20
Top AI models wrong on 57% of financial queries, 88% on complex multi-step questions — johncrickett · 2026-09-20
AI models err on 57% of financial queries; researcher blames marketing for over-reliance — gerardsans · 2026-09-21