← Back to feed

OpenAI audited SWE-Bench Pro — the agentic coding benchmark it had recommended as the replacement for SWE-bench Verified — and found approximately 30% of its 731 tasks have structural defects. The lab has retracted its endorsement.

The writeup, published under the title “Separating signal from noise in coding evaluations,” is both an indictment of the benchmark and a demonstration of why the situation is genuinely difficult to fix.

What the Audit Found

The audit used two parallel methods. Codex-based investigator agents inspected repository history, executed tests, and analysed failure traces at scale. Separately, five experienced software engineers reviewed each flagged task independently. Automated detection flagged 200 tasks (27.4% of the total); human review flagged 249 (34.1%). The two methods agreed on 74% of cases.

Four failure patterns dominated the flagged set:

• Overly strict tests — enforcing specific implementation details that were never mentioned in the prompt
• Underspecified prompts — hiding requirements from the problem description that only the hidden test suite knows about
• Low-coverage tests — incomplete fixes pass because the tests do not actually check the feature being evaluated
• Misleading prompts — pointing models toward incorrect behaviour, then penalising them for following the directions

The structural cause applies to all of them. SWE-Bench Pro tasks are sourced from real GitHub pull requests. PRs are written for human collaborators, not model evaluators. The test suite validates a specific contributor’s implementation approach, not whether the underlying problem was solved in any other valid way. A model that reaches the correct answer by a different path fails.

The Retraction

OpenAI had recommended SWE-Bench Pro explicitly as a response to SWE-bench Verified’s problems. Verified was flagged for contamination — models had likely encountered its tasks during training, inflating scores. SWE-Bench Pro was supposed to fix that with longer-horizon tasks from newer, harder-to-contaminate repositories.

The retraction statement: “Given the issues uncovered in this analysis, we retract our earlier recommendation to adopt SWE-Bench Pro.”

No replacement has been named. OpenAI says it is calling for “benchmarks built by experienced software developers specifically to test model capabilities” — tasks designed from the outset for evaluation, not repurposed from human-to-human engineering workflows.

The Meta-Layer

OpenAI used Codex-based agents to conduct the audit. The models now strong enough to post 80% on SWE-Bench Pro are also strong enough to inspect those same tasks in detail, identify failure modes, and categorise them at scale. OpenAI noted: “Evaluation flaws are easier to detect now than they would have been even a short time ago.”

That creates a loop. The benchmarks measuring model progress are now being audited by those same models. It also means score inflation is increasingly visible rather than latent. A 30% broken-task rate compresses the real signal range. A gain from 59% to 64% on a benchmark with 200 defective tasks is not the same as the same gain on a clean one.

Context for Current Leaderboard Numbers

Models posting high SWE-Bench Pro scores — Claude Fable 5 at 80.3%, Claude Mythos Preview at 77.8%, GPT-5.6 Sol at 64.6%, GPT-5.6 Terra at 63.4% — all achieved those numbers on the same compromised task set. The relative ranking is not necessarily wrong; the absolute numbers are less meaningful than they appeared.

SWE-bench Verified remains on the leaderboard at swebench.com. It has its own contamination concerns but has not been retracted. Terminal-Bench, DeepSWE, and Artificial Analysis’s composite Coding Agent Index are the current alternatives drawing most of the competitive attention.

The coding eval landscape has now cycled through two major benchmark collapses in under twelve months. The next credible replacement has not been announced.