FrontierCode Diamond: Claude Opus 4.8 Merges 13.4% of PRs Under Real Maintainer Standards
Cognition, the team behind Devin, published FrontierCode in August 2026 — a coding benchmark that asks a different question than SWE-bench. Not “does the code work?” but “would a real maintainer merge this?”
The answer, for every model tested, is mostly no.
What FrontierCode Measures
The benchmark grades code on mergeability: correctness, test quality, scope discipline, style, and adherence to codebase conventions. Passing requires clearing all “blocker” criteria that a maintainer would treat as hard stops. Anything that fails those receives a score of zero.
Tasks were written by 20+ open-source maintainers, each spending 40+ hours building and validating their problems. Every task was manually reviewed by a Cognition researcher. Cognition reports an 81% lower false positive rate compared to SWE-Bench Pro — meaning SWE-Bench was grading as passing code that human reviewers would reject.
Three difficulty tiers: Extended (150 tasks), Main (100 tasks), and Diamond (50 hardest).
Results
FrontierCode Diamond (50 tasks)
| Model | Pass Rate |
|---|---|
| Claude Opus 4.8 | 13.4% |
| GPT-5.5 | 6.3% |
| Gemini 3.1 Pro | 4.7% |
| Kimi K2.6 (best open-source) | 3.8% |
FrontierCode Main (100 tasks)
| Model | Pass Rate |
|---|---|
| Claude Opus 4.8 | 34.3% |
| Kimi K2.6 | 16.0% |
FrontierCode Extended (150 tasks)
| Model | Pass Rate |
|---|---|
| Claude Opus 4.8 | 51.8% |
| Kimi K2.6 | 37.0% |
The Gap Between Correct and Good
The headline number is sobering: the best model on the market at the time of testing fails 86.6% of its hardest coding tasks when judged by the standards of someone who would actually merge the PR.
On the cost side, GPT-5.5 uses up to 4x fewer tokens than Opus 4.8 on equivalent tasks while scoring only half as well. Depending on the use case, that tradeoff may favor GPT-5.5 for lower-stakes work.
The open-source gap is large. Kimi K2.6, the best open-weight model on Diamond, scores 3.8% — less than a third of Opus 4.8. On Extended, the gap narrows: 37% vs 51.8%, a more competitive spread.
Why This Matters
SWE-bench has driven a lot of benchmark competition over the past two years. Models routinely report 80-97% on Verified. FrontierCode’s Diamond set shows that number doesn’t translate directly into production-quality output.
The benchmark doesn’t test correctness in isolation. It tests correctness plus test quality, plus scope discipline, plus style. Code that passes unit tests but deletes half the project’s conventions to do so fails. That’s the actual definition of “good” in a professional codebase.
Cognition is selling Devin, so it has an interest in making the benchmark hard. But the methodology is transparent — 20+ maintainers, 40 hours per task, adversarial testing, multi-stage review. The underlying claim — that functional correctness benchmarks don’t capture real code quality — has been a known concern in the evaluation community for over a year. FrontierCode is the most rigorous attempt to close that gap.