OpenAI Retires SWE-bench Verified: 59% of Audited Tasks Have Broken Tests, Frontier Models All Saw the Answers
OpenAI has stopped reporting SWE-bench Verified scores and is recommending all frontier model developers do the same. The company published an analysis identifying two problems that, together, mean scores on the benchmark no longer reflect real-world software development capability.
The benchmark has been the de facto standard for measuring autonomous coding ability since August 2024. Nearly every model release from Anthropic, Google, xAI, and others has led with a SWE-bench Verified number. That metric is now functionally retired.
The Two Problems
Broken test cases. OpenAI audited a 27.6% subset of problems that models frequently fail and found that at least 59.4% of those problems have test cases that reject functionally correct submissions. Models were solving the underlying engineering problem, but getting marked wrong because the test itself was wrong. This is a dataset quality issue that OpenAI says persisted despite significant effort during the benchmark’s original creation.
Contamination. SWE-bench problems are sourced from open-source repositories that all major frontier model providers use for training. OpenAI tested whether models had seen the evaluation data before and found that every frontier model they tested could reproduce the gold patch, the original human-written bug fix used as the ground-truth answer, verbatim. The same applied to verbatim problem statement specifics in certain tasks.
The combination is structurally disqualifying. When a model scores 80% on SWE-bench Verified, it is unclear whether it solved 80% of the software problems or whether it memorised 80% of the answers and happened to match the broken tests.
Why Progress Stalled
OpenAI notes that progress on SWE-bench Verified has slowed considerably. The top score improved from approximately 74.9% to 80.9% over the last six months, despite substantial capability improvements across the frontier. That compression into a narrow band is consistent with a benchmark approaching saturation from the training-data side rather than genuine capability limits. At 80.9%, the remaining failures are increasingly explained by dataset problems rather than model limitations.
Implications for Benchmark Comparisons
SWE-bench Verified scores have driven a significant portion of the model comparison narrative over the past two years. The table below shows the numbers that dominated the conversation as of this writing:
| Model | SWE-bench Verified |
|---|---|
| Claude Fable 5 | 95.0% |
| Claude Mythos 5 | 95.5% |
| Claude Opus 4.8 | 88.6% |
| GPT-5.5 | 88.7% |
| Claude Opus 4.7 | 87.6% |
| Kimi K2.7 Code | 81.0% (est.) |
At the top, where Fable 5 and GPT-5.5 post numbers above 88%, the contamination issue is particularly acute. Both models were trained at massive scale on open-source code. The degree to which their high scores reflect capability versus memorisation is now explicitly unresolved.
SWE-bench Pro, which sources problems from private or recently-created repositories, is less affected. The same contamination argument applies with less force to tasks models could not have seen during training. That benchmark has become the more defensible comparison point for frontier-tier models.
What Comes Next
OpenAI says it is building new, uncontaminated evaluations. No timeline was given. In the interim, the company says it will not report SWE-bench Verified and recommends others follow. The company has previously introduced SWE-bench Pro and terminal-based benchmarks with stricter contamination controls.
The broader implication is that the industry’s preferred coding benchmark has been reporting optimistic numbers for some time. Real capability progress on autonomous software engineering continues, as demonstrated by tools like Claude Code and Codex that handle real production tasks. The benchmark just stopped being the reliable signal it appeared to be.