Ornith-1.5-397B Reaches 86% SWE-bench Verified with a Self-Generating Training Loop
Ornith-1.5-397B has reached 86% on SWE-bench Verified, placing first on the HuggingFace open-model leaderboard across 68 entries and at selfhost.directory’s tracker of 57 self-hostable models. The model is evaluated with the OpenHands harness, 256K context, no git history access, and no network access — conditions designed to prevent reward hacking.
The score is a 3.6-point gain over Ornith-1.0 (82.4%). The larger gains are in agentic tasks: Terminal-Bench 2.1 rises from 77.5% to 86.1%, and DeepSWE jumps from 8 to 56 — close to a total reconstruction of the model’s tool-use capability across versions.
The Training Method
Ornith-1.5 extends Ornith-1.0’s self-scaffolding framework into a full self-improvement loop with three stages per training cycle:
- Task generation. Given an environment or codebase, the model proposes tasks harder than what it has already solved.
- Scaffold generation. The model builds task-specific scaffolds: tools, decomposition strategy, evaluation logic.
- Rollout and reward. The policy generates solution rollouts; reward propagates back through all three stages.
The reward function multiplies three terms: validity (does the scaffold run and reliably evaluate solutions?), frontier difficulty (is the task solved roughly 20% of the time — not trivially, not impossibly?), and novelty (is it distinct from previously seen tasks?). A task that fails any criterion scores zero reward, preventing the model from repeating easy ground.
The loop compounds: as the model improves, the same system finds harder tasks, builds better scaffolds, and generates better training signal — without human curation of tasks or harnesses.
Benchmark Numbers
| Benchmark | Ornith-1.5-397B | Ornith-1.0-397B | DeepSeek V4-Flash | GLM-5.2 | Claude Opus 4.8 |
|---|---|---|---|---|---|
| SWE-bench Verified | 86.0% | 82.4% | 81.6% | 83.0% | 85.8% |
| SWE-bench Pro | 65.1% | 62.2% | 64.4% | 62.1% | 68.0% |
| SWE-bench Multilingual | 79.6% | 78.9% | 77.9% | 78.4% | 75.7% |
| Terminal-Bench 2.1 (Terminus-2) | 86.1% | 77.5% | 82.7% | 81.0% | 85.0% |
| DeepSWE | 56.0 | 8.0 | 54.4 | 46.2 | 59.0 |
| GPQA Diamond | 92.8% | 88.1% | 91.4% | 91.2% | 93.6% |
| MCP-Atlas | 80.0 | 76.4 | 74.6 | 77.8 | 82.2 |
| BrowseComp | 86.6% | 79.7% | 84.8% | 85.6% | 84.3% |
Kimi K3 (2.8T) scores 86.2% on SWE-bench Verified with the same OpenHands harness — marginally ahead — but sits outside the open-model leaderboard due to its proprietary license terms.
Family Breadth
The 1.5 release covers three scales:
- 397B MoE (24.4B active): Flagship, approximately 241 GB VRAM required
- 35B MoE (3B active): 79.0% SWE-bench Verified, beats Gemma 4-31B and Muse Glimmer-30B on agentic coding
- 9B Dense: 70.6% SWE-bench Verified, mobile-deployable with a quantized version confirmed to run on iPhone and Android
The 9B result is the notable outlier: it matches or exceeds Gemma 4-31B and Qwen 3.6-35B on SWE-bench despite being approximately 4x smaller by parameter count, at 70.6% versus those models’ 52.0% and 76.0% respectively on Terminal-Bench 2.1.
Caveats
All Ornith-1.5 results are self-reported by the DeepReinforce team, averaged over five independent runs, with evaluation methodology and anti-hacking conditions documented in the HuggingFace model card. Independent validation on the SWE-bench bash-only or vals.ai leaderboard has not yet appeared. Harness choice matters: OpenHands scores generally run higher than mini-SWE-agent bash-only runs on the same model, so the 86% figure is not directly comparable to proprietary-model scores sourced from vals.ai.