SWE-Rebench Exposes Contamination: DeepSeek V3 Scores Drop 46% on Fresh Tests
A growing body of results from SWE-rebench — a benchmark that uses post-training-cutoff GitHub pull requests so models cannot have seen the answers — is putting a hard number on an open secret: benchmark contamination is inflating AI coding scores, and Chinese labs are showing the largest gaps.
The Numbers
Results from the SWE-rebench V1 leaderboard comparing identical models on SWE-bench Verified versus SWE-rebench (Mar–Apr 2025 slice):
| Model | SWE-bench Verified | SWE-rebench | Drop |
|---|---|---|---|
| DeepSeek-V3-0324 | 39.7% | 21.3% | –18.4 pts |
| DeepSeek-V3-1226 | 35.2% | 21.9% | –13.3 pts |
| LLaMA-4-Maverick | 16.0% | 12.2% | –3.8 pts |
| Qwen2.5-72B | 11.3% | 9.3% | –2.0 pts |
| GPT-4.1 | 31.1% | 26.7% | –4.4 pts |
DeepSeek-V3-0324 scored 39.7% on SWE-bench Verified — competitive with mid-tier Claude and GPT releases — and 21.3% on SWE-rebench. That is a 46% collapse in absolute terms. Western models show a smaller but still real drop; nearly every model evaluated scores lower on decontaminated tasks, consistent with some incidental contamination across large web-scale training corpora. The difference is magnitude.
How the Benchmark Works
SWE-bench, introduced by Princeton researchers in 2023, uses real GitHub issues from popular Python repos to test whether a model can generate a passing patch. The problem: most of those issues were filed and resolved before any frontier model’s training cutoff. A model that saw the repos — including the merged PR, the discussion thread, and the fix — during training is not solving the task; it is retrieving a known answer.
SWE-rebench addresses this with temporal filtering. Tasks are drawn from GitHub issues filed and resolved after each model’s stated training cutoff. No model can have seen the patch during training. The V2 paper, published March 2026, scales this to 32,000+ tasks across 20 programming languages from 3,600+ repositories.
Execution is concrete: a patch either makes the test suite pass or it does not. There is no partial credit for “looks close.”
What the Drop Means
The contamination question matters beyond leaderboard rankings. Enterprise teams picking a coding agent are making budget and architecture decisions based on SWE-bench numbers. If those numbers include 10–18 points of memorised-answer credit, the economic case for some Chinese models looks materially different on real workloads.
The finding also has direct implications for how AI labs are evaluated in security-sensitive contexts. Models with inflated coding scores appear more capable of autonomous software modification than they actually are — which affects both threat models and procurement decisions.
A companion benchmark result on SWE-bench Verified further illustrates the ceiling problem: Qwen3-Max-Thinking achieved 75.3% on SWE-bench Verified, an impressive-sounding number that has not yet been independently replicated on decontaminated tasks.
The Fix
SWE-rebench is a moving target by design. Because the benchmark is refreshed monthly from fresh GitHub activity, any model that tries to pre-train on its tasks faces a constantly changing evaluation surface. That is the correct architecture for benchmarks where contamination risk is high — and it represents a template for how other high-stakes benchmarks should be designed.
For now, SWE-rebench results suggest that several Chinese lab models sit roughly 13–18 points lower on genuine coding ability than their SWE-bench Verified headline numbers imply.