OpenAI Sweeps GDP.pdf's Top Seven Slots — The Benchmark That Gives No Partial Credit
Artificial Analysis has added GDP.pdf to its public evaluation suite, and the leaderboard does not look like any other benchmark in the ecosystem.
GPT-6 Astra holds first place at 32.2% all-pass rate. GPT-6.1 Sol follows at 32.0%. Seven of the top eight positions belong to OpenAI models. The first Anthropic entry — Claude Opus 5.5 at high effort — sits at 28.8% in eighth place.
That result is worth pausing on. On the AA Intelligence Index, updated just two days ago, Anthropic holds the top four positions. On GDP.pdf, it holds none of the top seven.
What GDP.pdf Measures
GDP.pdf is a benchmark from Surge AI, now independently implemented and evaluated by Artificial Analysis. The design brief: 100 tasks drawn from ten professional domains, grounded in 4,592 pages of real-world source documents, scored against 1,275 atomic grading criteria. Models receive five independent attempts per task. GPT-5.6 Luna Medium acts as the judge.
The headline metric is all-pass rate: the fraction of attempts where every criterion passes. Not most criteria. All of them.
That is different from how most AI benchmarks work. If a model satisfies 10 of 11 criteria on a medical document task, that attempt counts as a failure. The all-pass structure is how professional review actually works — a contract with one incorrect clause is still an incorrect contract.
Artificial Analysis publishes an example. A healthcare task asks a model to describe EDS subtypes associated with TNXB gene mutations. Gemini 3.5 Flash passed 10 of 11 criteria — a 90.9% criterion rate — but failed the task entirely. The failing criterion: the model incorrectly classified the contiguous gene syndrome CAH-X as a distinct EDS subtype. That distinction matters clinically. It was wrong. The task failed.
The Leaderboard
| Model | Effort | All-pass Rate |
|---|---|---|
| GPT-6 Astra | xhigh | 32.2% |
| GPT-6.1 Sol | high | 32.0% |
| GPT-6.1 Sol | xhigh | 31.8% |
| GPT-6 Astra | max | 31.0% |
| GPT-6 Astra | high | 31.0% |
| GPT-6.1 Sol | max | 31.0% |
| GPT-6 Astra | medium | 30.4% |
| Claude Opus 5.5 | high | 28.8% |
| Claude Fable 5.1 | high | 26.8% |
| Claude Opus 5.5 | xhigh | 26.6% |
| Muse Spark 1.3 | max | 26.6% |
| Claude Fable 5.1 | max | 26.2% |
| Claude Opus 5.5 | max | 26.2% |
| Claude Sonnet 5.5 | max | 25.8% |
Source: Artificial Analysis GDP.pdf leaderboard, 6 October 2026. 205 configurations evaluated.
Surge AI’s own implementation uses a different harness and shows GPT-6 Astra at 34.2% under max reasoning — higher than AA’s 32.2% for the same model. The implementations are not directly comparable, and Artificial Analysis notes this explicitly. The AA figures represent an independent third-party run on the public dataset.
The Effort Effect
Neither OpenAI nor Anthropic models show a clean correlation between inference effort and all-pass rate on GDP.pdf.
For GPT-6.1 Sol, high effort (32.0%) outperforms both xhigh (31.8%) and max (31.0%). For Claude Opus 5.5, the same pattern holds: high (28.8%) beats xhigh (26.6%) and max (26.2%). Extended thinking budgets do not help on professional document work the way they help on mathematics or multi-step reasoning.
The hypothesis is structural. Satisfying granular factual criteria depends on accuracy and groundedness, not extended chain-of-thought. More thinking steps may introduce hedging, reframing, or subtle reclassifications that break criteria the model would have satisfied on a direct first pass.
The 32% Ceiling and What It Means
The best published score on the AA implementation is 32.2%. On Surge’s own harness it reaches 34.2%. In either case, more than two-thirds of professional document tasks result in complete failure even for the best available models.
That number sets a realistic baseline for AI deployment in document-heavy professional workflows. Healthcare, legal, finance, and compliance work involves exactly the kind of criteria-dense review that GDP.pdf benchmarks. A model that fails 68% of tasks completely — regardless of how well it performs on the other 32% — is not yet adequate as a standalone professional document agent.
It is adequate as a first-pass reviewer, a drafter, a collaborator. The benchmark does not show AI is useless in professional contexts. It shows where the accuracy floor sits.
Sol vs. Astra
GPT-6.1 Sol’s GDP.pdf performance at $2 per million input tokens versus GPT-6 Astra at $4 to $10 per million is the most directly actionable number on the leaderboard. Sol at high effort scores 32.0% — 0.2 points behind Astra at xhigh, within measurement noise. For cost-sensitive deployments running GDP-style professional document tasks at volume, the case for Astra over Sol weakens considerably on this benchmark.
Why This Does Not Match the Intelligence Index
The AA Intelligence Index weights a composite of reasoning, coding, and agentic capability benchmarks. GDP.pdf is a narrow, domain-specific professional document test. Models can lead on broad intelligence while underperforming on precise factual compliance. Anthropic’s September cluster was optimised for reasoning depth and agentic coding; OpenAI’s GPT-6 family has consistently led on retrieval-grounded factual tasks.
These are not contradictory findings. They are different capability dimensions. Buyers whose workloads look like professional document review should weight GDP.pdf more heavily than the Intelligence Index. Buyers whose workloads look like software engineering, mathematical reasoning, or long-horizon research should weight them inversely.