Real-SWE Launches: Frontier Models Top Out at 38.8% on Private Enterprise Codebases
Specific Labs released Real-SWE in September 2026: a coding benchmark built entirely from licensed private production codebases, where no task exists anywhere on the public internet.
The dominant public coding benchmarks draw from public GitHub repositories that models may have trained on. Real-SWE takes a different approach entirely. Tasks are sourced from private production code — billing systems, fintech platforms, enterprise AI sales tools — licensed directly from real companies and never published. Agents must navigate proprietary architecture with no prior exposure and no shortcut through memorisation.
Leaderboard
| Rank | Model | Agent | Resolution Rate |
|---|---|---|---|
| 1 | Fable 5.1 | Claude Code | 38.8% |
| 2 | GPT-6 Astra | Codex CLI | 33.8% |
| 3 | Gemini 3.8 Flash | Gemini CLI | 31.2% |
| 4 | GLM 5.3 | Claude Code | 28.8% |
| =5 | Grok 4.6 | Grok Build | 23.8% |
| =5 | Muse Spark 1.3 | Muse Code | 23.8% |
| 7 | Kimi K3 | Kimi Code | 18.8% |
| 8 | GPT-5.6 Sol | Codex CLI | 16.2% |
Resolution rate is pass@1 averaged over eight independent runs per task. Real-SWE evaluates model-and-agent pairings, not models in isolation.
Why scores compress
The gap below public benchmarks is not contamination alone. Real-SWE tasks are structurally harder across three dimensions:
Scope. The median reference solution touches 11 files, versus 6 in FrontierCode and DeepSWE. A single billing fix might span a NestJS API, a tax authority integration, a ledger write, and a VAT registration display layer — all in one task.
Business logic. Tasks require understanding company-specific rules and external services together. One sample task: fix invoice billing so each business charges the right tax, exempt customers aren’t taxed, VAT registrations appear on EU invoices, and every settled invoice files back to the tax authority under its own number. The prompt is underspecified by design — about par with DeepSWE and Terminal-Bench — because real engineering instructions are.
Infrastructure breadth. Task environments include AWS emulators, Docker, Kubernetes, GitHub, Linear MCP, PostgreSQL, MySQL, MongoDB, Redis, Go, Python, Node.js, Vitest, Slack, Intercom, Google Drive, and email. Agents get only the services a given workflow needs.
Time allocation is not the bottleneck. 71.4% of rollouts under 10 minutes failed; 73.4% of longer rollouts failed. The block is comprehension of the codebase, not available wall time.
Notable results
Fable 5.1’s lead is narrow. GPT-6 Astra trails by 5 points (33.8% vs 38.8%), and Gemini 3.8 Flash places third at 31.2% — within a plausible statistical range of Astra given the eight-run averaging methodology.
GLM 5.3 from Zhipu AI places fourth at 28.8%, using Claude Code as its agent. The result suggests the agentic harness matters as much as the underlying model: GLM 5.3 running Claude Code beats Grok 4.6 and Muse Spark 1.3 running their own native agents.
GPT-5.6 Sol finishes last at 16.2%, 22 points behind Fable 5.1. That is a significant gap given Sol’s top-tier benchmark scores on SWE-bench Verified and LiveBench. Score compression at the frontier does not appear uniform across model families when the codebase is private and the tasks are economically consequential.
Context for AI coding evaluation
Real-SWE is the latest entrant in a growing challenge to SWE-bench hegemony. The last twelve months have seen OpenAI retire SWE-bench Verified after auditors found 59% of tasks had broken tests, OpenAI retract SWE-bench Pro after its own review flagged 30% of tasks as broken, and a wave of alternative benchmarks — Senior SWE-bench, FrontierCode, DeepSWE — each trying to shift evaluation toward realistic, harder tasks.
Real-SWE’s differentiator is provenance: the code is private by origin, not by curation. That is a harder guarantee to fake and a harder benchmark to overfit to.
Specific Labs selected codebases from companies with material usage: a Luma/Partiful competitor with 200K+ users, a fintech platform processing 100K+ bank statements, and enterprise AI sales tools supporting complex workflows. The stated selection criterion is that the code was written to meet a real user or business need, not to create a benchmark task.