DeepSeek V4 Pro Scores 96.4% on SWE-Bench but Only 77.4 on LiveBench — the Specialist Model Problem
DeepSeek V4 Pro 0813 sits at 96.4% on SWE-bench Verified, the second-highest score ever recorded on that leaderboard, two tenths behind Claude Opus 5. On LiveBench, it scores 77.4 overall — twelfth on the current leaderboard, behind Kimi K3 (79.2), Qwen 3.8 Max (78.5), Grok 4.6 (78.0), and eight other models with lower SWE-bench scores.
The gap between these two rankings is not an anomaly. It describes what each benchmark actually tests.
Two Different Questions
SWE-bench Verified presents software engineering tasks: given a GitHub issue and a codebase, submit a patch that resolves the issue. The benchmark was designed by software engineers to test software engineering ability. It is good at what it tests. A model that excels on SWE-bench is demonstrably capable at autonomous software repair in a structured environment.
LiveBench evaluates seven categories: reasoning, coding, agentic coding, mathematics, data analysis, language, and instruction following. Questions are sourced from recent research papers and news to limit contamination. Results are scored against objective ground truth. It is a generalist benchmark designed to resist gaming.
DeepSeek V4 Pro 0813’s scores across those two frames tell different stories about the same model.
The Current Rankings
On SWE-bench Verified (vals.ai neutral harness, August 13):
| Rank | Model | SWE-bench |
|---|---|---|
| 1 | Claude Opus 5 | 97.0% |
| 2 | DeepSeek V4 Pro 0813 | 96.4% |
| 3 | GPT-5.6 Sol | 96.2% |
| 4 | Grok 4.6 | 95.6% |
On LiveBench overall (current leaderboard):
| Rank | Model | LiveBench | SWE-bench |
|---|---|---|---|
| 1 | Claude Fable 5 Max Effort | 83.0 | 95.0% |
| 2 | GPT-5.6 Sol Max Effort | 81.0 | 96.2% |
| 3 | GPT-5.5 Thinking xHigh | 80.2 | 88.7% |
| 4 | Claude Opus 5 Thinking | 80.1 | 97.0% |
| 5 | Smaug-Agentic (open) | 79.5 | — |
| 6 | Kimi K3 (open) | 79.2 | 93.4% |
| ~12 | DeepSeek V4 Pro 0813 | 77.4 | 96.4% |
Kimi K3, with a 93.4% SWE-bench score, ranks above DeepSeek V4 Pro on LiveBench by 1.8 points despite having a 3-point lower SWE-bench score. Claude Fable 5 at 95.0% SWE-bench leads the LiveBench overall ranking. GPT-5.6 Sol at 96.2% SWE-bench is a clear second on LiveBench. These models sit at the intersection of both rankings; V4 Pro does not.
Why the Gap Exists
DeepSeek designed V4 Pro with coding as a primary capability emphasis. Its April 2026 tech report, post-training approach, and public benchmark positioning all reflect this. The model’s 96.4% SWE-bench result on a neutral harness validates that the training investment landed. The LiveBench shortfall is the corresponding cost: optimising heavily for one capability domain leaves residual gaps in others.
That trade-off is visible at a model-family level too. The default DeepSeek V4 Pro (without the 0813 update) scores 71.6 on LiveBench — 5.8 points below the 0813 version. The update improved performance across the board, but V4 Pro still trails models from Anthropic, OpenAI, xAI, Moonshot, and Alibaba that were trained with broader multi-domain coverage from the start.
The Arena Angle
Arena.ai added deepseek-v4-pro-high to its Agent Arena and text leaderboards on August 19, six days after the vals.ai SWE-bench update. Human preference evaluations tend to surface different capability gaps than objective benchmarks — particularly in instruction following, response quality, and reasoning legibility. Where V4 Pro-High lands on Arena ELO rankings will be the next meaningful data point for practitioners weighing coding specialisation against general deployment.
What It Means for Benchmark Interpretation
SWE-bench Verified is now the dominant signal in the AI coding agent market. Labs publish it, users cite it, investors reference it. It earns that attention — it is a hard, well-curated benchmark that correlates with real-world software engineering capability. But it is one dimension.
A model at 96.4% on SWE-bench and 77.4 on LiveBench is a strong choice for autonomous software repair tasks and a weaker choice for general-purpose reasoning, data analysis, or instruction following at the same price point. A model at 93.4% on SWE-bench and 79.2 on LiveBench is a different trade-off.
Neither is wrong. But the benchmark cited in the press release is the one optimised for. That asymmetry is worth holding onto when reading the next announcement.