FrontierCode 1.1 Lands on Artificial Analysis: Best Models Still Fail 42% of Real Merge Reviews
Artificial Analysis has added FrontierCode 1.1 as a standalone leaderboard, independently benchmarking current frontier models against a standard that matters to actual software engineers: would a maintainer merge this?
Cognition built FrontierCode with the maintainers of 36 real open-source repositories, spending more than 40 hours crafting each of the benchmark’s 150 tasks. AA runs the 100-task Main subset, three attempts per task, using its open-source Stirrup harness capped at 250 turns per attempt. GPT-6 Luna (medium) serves as the LLM judge.
Key Numbers
| Model | Score | Pass Rate |
|---|---|---|
| Claude Opus 5.5 (max+fallback) | 52% | 58% |
| GPT-6.1 Sol (max) | 49% | — |
| Claude Fable 5.1 (max+fallback) | 49% | — |
| MiMo-V2.6-Pro (max) | 45% | — |
| DeepSeek V4.1 Flash (max) | 44% | — |
| Kimi K3 (max) | 44% | — |
| Muse Glimmer (high) | — | 16% |
Nine models from six labs sit between 43% and 52%. The top of the leaderboard is unusually compressed: the gap between first and ninth place is 9 percentage points.
What the Benchmark Actually Measures
FrontierCode goes beyond test passage. Each attempt is also graded on scope discipline: excessive diffs, modifying files outside the relevant scope, and adding unhelpful tests all count against a model. The verifier applies deterministic hidden tests plus an LLM rubric review. A single blocking failure scores the attempt zero regardless of anything else it got right.
9 of the 100 tasks were not passed by any model on any attempt. No task was passed by every model every time. This is a benchmark the frontier hasn’t solved.
Cost-Efficiency Standout
GPT-6.1 Sol (max) at $0.81 per task matches Claude Fable 5.1 at 49% while costing less than a tenth of what the leading Anthropic models spend per task. MiMo-V2.6-Pro, from Xiaomi, is the highest-scoring open-weights model at 45%. DeepSeek V4.1 Flash and Kimi K3 both hit 44% at costs substantially below the closed-API frontier.
Why This Matters
Most coding benchmarks measure test passage. FrontierCode measures whether a change would actually land in a real codebase. The distinction is significant: the same model that passes all tests can still produce a diff a maintainer would reject for being too large, touching the wrong files, or missing the maintainer’s unstated standards.
The benchmark is not yet part of AA’s Coding Agent Index but is planned for inclusion. Tasks remain private to prevent contamination. AA’s Stirrup harness ensures all models run under identical conditions for direct comparison.