SWE-bench Verified Adds Bash-Only Track — Fable 5.1, GPT-6 Astra, and Muse Spark 1.3 Are Missing From It
SWE-bench Verified now offers two distinct views of the coding leaderboard: the full custom-agent track, where teams bring their own scaffolding, RAG pipelines, and multi-rollout review systems, and a new bash-only track that evaluates every model using the same minimal harness. The second track uses mini-SWE-agent — a plain ReAct loop running in a bash shell with no tools and no special scaffold structure. The goal is an apples-to-apples model comparison that isolates raw language model capability from engineering infrastructure.
The bash-only scores from neutral harnesses tell a clean story at the top. Claude Opus 5 sits at 97.0% on vals.ai’s neutral bash-only evaluation, setting the ceiling. DeepSeek V4 Pro follows at 96.4%, GPT-5.6 Sol at 96.2%, Grok 4.6 at 95.6%, and Claude Fable 5 at 95.0%. The cluster between 88 and 97 percent represents the prior generation of frontier models, evaluated on a consistent harness.
The Blank Rows
Three models launched since August 2026 are absent. Claude Fable 5.1, GPT-6 Astra, and Muse Spark 1.3 have no SWE-bench Verified scores on any neutral bash-only harness. Fable 5.1 is Anthropic’s current intelligence leader on Artificial Analysis and tops LiveBench at 83.4 overall. GPT-6 Astra is OpenAI’s multimodal frontier model, now on all three Arena leaderboards. Muse Spark 1.3 from Thinking Machines is fourth on LiveBench at 81.6. None have Verified results.
What the community has instead:
| Model | LiveBench Agentic Coding | Terminal-Bench 4.0 | DeepSWE 1.1 |
|---|---|---|---|
| Claude Fable 5.1 | 66.1 | 55.8% | — |
| GPT-6 Astra | 57.3 | 57.9% | 74.1% |
| Muse Spark 1.3 | 64.1 | — | — |
These are auxiliary benchmarks run under different harnesses and task distributions. They do not map directly to SWE-bench Verified scores and cannot be compared across models on an equal footing.
Why the Gap Exists
SWE-bench Verified reached the point where task saturation became a concern. Scores above 95% make differentiation difficult, and the benchmark’s task set dates to an era when frontier models scored well below that range. After a contamination audit in mid-2026 raised questions about test integrity, labs shifted evaluation attention to newer benchmarks: SWE-bench Pro (which shows a 16-point spread where Verified shows 3.6), Real-SWE (private enterprise codebases, top score 38.8%), and FrontierCode 1.1 (53.3% for GPT-6 Astra at last check).
That shift is rational but creates a gap. Without a shared Verified score, comparing Fable 5.1 against Opus 5 on coding capability requires stitching together numbers from different benchmarks with different methodologies. The bash-only mini-SWE-agent track would resolve this — if anyone runs it.
What the Bash-Only Track Is Actually For
The value of the bash-only mode is that it eliminates the scaffold variable. On the full SWE-bench Verified leaderboard, scores reflect a combination of model capability and scaffold quality. The same base model can score materially differently depending on whether the harness uses single-pass completion, multi-rollout, or RAG augmentation. The bash-only track removes that noise.
As of September 2026, the top bash-only scores cluster tightly between 93 and 97 percent for the 2025-vintage frontier models. The gap between ranks one and five is 2 percentage points. That compression is itself a signal: at the current ceiling, Verified no longer discriminates between top-tier models even in the cleanest comparison mode.
The introduction of the bash-only track was the right call. Getting the newest generation of models into it would make it matter again.