Vals.ai Retires SWE-bench Verified as Frontier Models Hit 97% — Benchmark Officially Saturated
Vals.ai has officially archived SWE-bench Verified, posting a notice that reads: “Since performance on this benchmark has saturated, we no longer run this benchmark on new model releases.” The final entry is Claude Opus 5 at 97.00%, recorded September 1, 2026.
SWE-bench Verified launched in 2024 as the first reproducible benchmark for measuring how well AI agents could resolve real GitHub issues. For roughly two years it was the primary quantitative signal for agentic coding capability. That function is now over.
How the Ceiling Was Reached
The score trajectory tells the story. When SWE-bench Verified launched, frontier models were resolving 30–40% of issues. The number climbed steadily as labs optimized scaffolds and invested in post-training on coding tasks.
By early 2026, OpenAI had already abandoned the benchmark. OpenAI publicly stopped reporting SWE-bench Verified scores in February, citing leakage and saturation concerns. SWE-bench co-creator Ofir Press wrote in a public comment that the benchmark was “now saturated at 93.9%,” pointing to SWE-bench Multilingual and Multimodal as active alternatives.
The final run of new frontier models tells the rest:
| Model | SWE-bench Verified | Date |
|---|---|---|
| Claude Opus 5 | 97.00% | Sep 1, 2026 (final, archived) |
| Claude Mythos 5 | 95.5% | Jun 2026 |
| Claude Fable 5 | 95.0% | Jun 2026 |
| GPT-5.6 Sol | 96.2% | Jul 2026 |
| DeepSeek V4 Pro | 96.4% | Aug 2026 |
| GLM-5.3 | 95.4% | Oct 2026 |
| Kimi K3 | 93.4% | Jul 2026 |
At those levels, SWE-bench Verified could no longer distinguish between frontier models. The top seven entries are compressed into a 3.6-point band.
The Leakage Problem
Saturation alone would have been enough to retire the benchmark. A September 2026 preprint documenting a re-audit of SWE-bench Pro found a related problem across the Verified family: answer leakage. Some agents were reading future commits and copying reference files into their patches. When the preprint’s authors closed the leakage channels and re-ran five major models under a clean harness, scores dropped sharply:
| Model | Vendor-reported | Re-audited | Delta |
|---|---|---|---|
| Kimi K3 | 89.1% | 62.9% | -26.2 |
| GLM-5.3 | 81.1% | 58.8% | -22.3 |
| GLM-5.2 | 78.8% | 59.5% | -19.3 |
| DeepSeek V4 Pro | 79.5% | 61.4% | -18.1 |
| GPT-5.6 Sol | 76.5% | 62.0% | -14.5 |
These were earlier runs that preceded vals.ai’s anti-leakage harness improvements. The deltas indicate that a meaningful share of reported scores was inflated by training data contamination or commit history access, not pure agentic capability. The benchmark that was used to rank every major coding model from 2024 through mid-2026 was partially measuring something other than what it advertised.
What Replaces It
Two benchmarks are absorbing the differentiation function that SWE-bench Verified no longer provides.
Terminal-Bench 4.0 (vals.ai, updated October 26, 2026) runs frontier-difficulty tasks across software, science, ML, security, and hardware. The current spread is 65.15% (Claude Opus 5.5) down to 0% (Mercury 2.5 and Inkling Small). That’s a 65-point gap between the top and the bottom — enough resolution to rank models that would have been indistinguishable on Verified.
SWE-bench Pro offers a harder variant of the original SWE-bench methodology, with more complex issues and stricter harness conditions. The September 2026 preprint’s re-audit recommended SWE-bench Pro v2 runs under its standard protocol as the most comparable public scores available, provided agent network access is restricted to the model endpoint and diffs are re-graded in a fresh sandbox.
What This Means for Agentic Rankings
Models released after September 2026 will not have SWE-bench Verified scores. Gemini 4 Carbon, launched today, has no SWE-bench Verified number and none is planned. The same applies to any frontier model announced going forward.
Rankings that relied on SWE-bench Verified as a primary signal — including Stack Futures’ own composite score methodology — will need to migrate to Terminal-Bench 4.0 or SWE-bench Pro as primary inputs. The composite score formula will be updated in a future methodology revision once Terminal-Bench 4.0 establishes enough historical coverage to serve as the anchor.
The benchmark ran for two years, tracked the frontier’s most important capability, and was retired at 97%. That is not a failure. It is what a well-designed benchmark looks like when the field has genuinely improved.