← Back to feed

Artificial Analysis has launched AA-AnalystAgent, a standalone agentic benchmark covering quantitative analysis on real-world spreadsheets and documents. The headline finding: the best model in the field — Claude Opus 5 — gets only 54% of tasks right across five consecutive attempts. GPT-5.5 trails at 50%. Claude Fable 5 sits at 49%.

The benchmark runs 80 tasks across 14 business and scientific domains: healthcare expenditure reports, trade statistics, hydrology data, government appropriations, energy cost models, financial models, environmental reporting, and project schedules. Work is categorised into five functional buckets that map to real analyst workflows — source lookup and diagnosis, filter and total, ratios/trends/sensitivities, P&L modelling, and cash/balance-sheet/valuation modelling.

The pass^5 Methodology

The key design decision is pass^5 scoring. Each task is run five times per model. A task only counts as solved when the model answers correctly on all five independent attempts. The rationale: if an AI analyst gives you an answer you still have to verify, it has not done the work. Only repeatable accuracy removes the verification burden from the human.

This is meaningfully different from pass@1 (average single-attempt accuracy) and pass@5 (whether the model gets it right at least once across five runs). The gap between these metrics is where the leaderboard gets interesting.

Reliability Separates the Top Models

GPT-5.5 (xhigh) posts the highest pass@1 at 66%. Gemini 3.1 Pro Preview reaches 64% on single attempts. Claude Opus 5 is also at 64% on pass@1. But on pass^5, Opus 5 leads at 54%, GPT-5.5 drops to 50%, and Gemini 3.1 Pro falls to 41% — finishing ninth overall.

Gemini 3.1 Pro solves 81% of tasks at least once, but only 41% on all five. The model is capable; it is not consistent. AA’s framing is direct: an answer that is right at random still needs verification, which defeats the purpose.

Among open-weight models, Kimi K3 (max) leads at 39% pass^5 — 15 points behind the closed frontier.

How Models Fail

AA classified 1,567 failing attempts across ten leading models into seven failure modes. The most widespread: anchoring on a wrong early interpretation, present in 57% of failures. Models commit to a source or methodology early and defend it through the rest of the trajectory even when evidence contradicts it.

The failure profiles diverge by lab. Gemini 3.1 Pro trusts its sources but makes execution errors — modelling, scaling, and aggregation mistakes account for 51% of its failures, with skipped verification in 39% of cases. Grok 4.5 (high) reads domain terms accurately but then overrides what the documents say with its own priors, substituting assumptions for evidence.

Cost Range

Price variation across the leaderboard is extreme. Claude Sonnet 4.6 and Xiaomi MiMo-V2.5-Pro both score 20% on pass^5. The cost to reach that score: $1.34 per task for Sonnet 4.6, $0.05 per task for MiMo. A 27x cost difference for identical output quality.

AA-AnalystAgent uses a privately held question set to limit contamination risk. The benchmark is not part of the AA Intelligence Index. The full methodology, including the agent harness (Stirrup, open-source), toolset, and grading prompts, is documented on the AA methodology page.