SWE-Bench Pro Differentiates Where Verified Cannot: 16-Point Spread vs 3.6
Seven models now score above 93% on SWE-bench Verified. Four exceed 95%. The spread from Kimi K3 (93.4%) to Claude Opus 5 (97.0%) is 3.6 percentage points. At that density the benchmark cannot meaningfully rank the frontier — and the field’s habit of citing SWE-bench Verified scores as the primary coding capability signal is becoming a problem.
SWE-bench Pro, the harder variant, already exists. It is not widely tracked. It should be.
The Verified Cluster
Current standings on SWE-bench Verified (vals.ai neutral bash-only harness):
| Rank | Model | Verified |
|---|---|---|
| 1 | Claude Opus 5 | 97.0% |
| 2 | DeepSeek V4 Pro 0813 | 96.4% |
| 3 | GPT-5.6 Sol | 96.2% |
| 4 | Grok 4.6 | 95.6% |
| 5 | Claude Mythos 5 | 95.5% |
| 6 | Claude Fable 5 | 95.0% |
| 7 | Kimi K3 | 93.4% |
Four labs. Seven models. All within 3.6 points of each other. When GPT-5.6 Sol at 96.2% and Claude Fable 5 at 95.0% are separated by 1.2 percentage points — and both results carry statistical uncertainty — the leaderboard position is effectively noise.
In April 2026, when GPT-5.5 scored 88.7% and Claude Opus 4.7 scored 87.6%, those numbers described a real gap. The benchmark had headroom. Now it does not.
The Pro Numbers Tell a Different Story
SWE-bench Pro presents harder, longer-horizon engineering tasks. For the three models with reported Pro scores:
| Model | Verified | Pro | Drop |
|---|---|---|---|
| Claude Fable 5 | 95.0% | 80.3% | -14.7 pts |
| Claude Opus 5 | 97.0% | 79.2% | -17.8 pts |
| GPT-5.6 Sol | 96.2% | 64.6% | -31.6 pts |
Three observations.
The spread is 16 points, not 3.6. Fable 5 at 80.3% versus GPT-5.6 Sol at 64.6% is a decisive gap for any engineering team choosing an autonomous coding tool. On Verified, those same models are 1.2 points apart and indistinguishable by any reasonable standard.
The ranking reverses. On Verified, Opus 5 leads Fable 5 by 2 percentage points. On Pro, Fable 5 leads Opus 5 by 1.1 points. The model that “wins” the familiar leaderboard loses on the harder one. This matters for anyone using Verified scores to pick a coding agent.
GPT-5.6 Sol drops the most. From 96.2% on Verified to 64.6% on Pro — a 31.6-point fall, nearly twice the drop of either Anthropic model. The pattern suggests Sol is tuned for the task structure and problem complexity of Verified in ways that do not generalise to Pro’s longer-horizon tasks.
What Pro Actually Tests
SWE-bench Verified uses 500 human-filtered GitHub issues. The tasks are solvable, the descriptions are clear, and the test patches are correct. It is a clean, well-designed benchmark — for 2024-era models. When frontier models were resolving 50-60% of tasks, 500 problems provided ample signal.
SWE-bench Pro raises the difficulty across three axes: problem complexity, solution length, and reasoning chain requirements. Models that succeed on Pro need to plan multi-file edits, reason across larger codebases, and maintain coherent state over longer context spans. Those are engineering tasks. The Verified tasks, at 95%+ resolve rates, are closer to autocomplete.
The Senior SWE-bench variant goes further — real-world engineering tasks with ambiguous requirements and cross-component dependencies. Frontier models collectively cap around 24% on Senior. That headroom is enormous.
The Practical Implication
Any engineering team evaluating coding agents on Verified-only scores is getting a misleading picture. The 1-2 point differences between top models on Verified are inside the noise. The 16-point differences on Pro are not.
The industry is slow to shift because Verified has a long citation history, well-established harnesses, and broad lab participation. Pro does not yet have the same infrastructure. But the data is there. As Verified approaches ceiling — Opus 5 is already at 97.0% against a practical maximum of around 98-99% — the field will have to move.
The harder benchmark is already built. The question is when the labs start racing on it.