AI Chatbots Give Wrong Financial Answers 57% of the Time — Claude Opus 5 Is Best at 39%
Tech company Saturn ran 18 AI models through 121 financial questions and found them wrong 57% of the time on average, according to a Financial Times report citing the study. The result holds across the leading chatbots — ChatGPT, Claude, Copilot, Grok, and Gemini — with no model clearing the bar for reliable financial guidance.
The error rate climbs sharply on harder questions. When asked complex queries involving multiple calculations — the type of question a retail investor might ask about compound growth, tax-adjusted returns, or pension projections — Google’s Gemini 3.5 Flash and Claude Haiku 4.5 gave wrong answers 99% of the time. ChatGPT 5.6 Luna missed 58% across all question types.
The Range
Claude Opus 5 with reasoning mode was the top performer, wrong 39% of the time. That is the best result in the dataset and still means the model gives a wrong financial answer roughly two out of every five queries.
The gap between models is meaningful. Reasoning-enabled flagship models perform considerably better than the smaller, faster tiers. But even at the top of the range, the error rate is high enough that any financial professional using these tools without independent verification is taking on systematic risk.
A separate PensionBee consumer survey of 1,000 US adults found that 57% said they would follow AI financial advice without independently verifying it, suggesting the deployment risk compounds on both sides: models that are frequently wrong, and users who are unlikely to check.
Why It Matters
Financial queries sit at the intersection of where AI is being most aggressively deployed and where errors have real consequences. Banks, wealth managers, and fintech platforms are integrating frontier models into customer-facing tools. The Saturn numbers suggest the models as tested are not reliable enough for unsupervised use in that context.
The harder finding is structural: the models that are cheap and fast enough for scaled deployment — Gemini 3.5 Flash, Haiku 4.5 — are also the ones with the highest error rates on complex calculations. The model that performs best — Claude Opus 5 reasoning — costs orders of magnitude more per query and has latency unsuited to conversational interfaces.
That gap is unlikely to close quickly. Financial calculation accuracy requires precision that differs from the probabilistic pattern-matching that drives general language quality. The models producing fluent, confident-sounding responses to financial questions are not necessarily the ones computing the right answer.