Terminal-Bench Launches Month-Scale Challenges: Claude Opus 4.8 Ran 12 Hours and Failed
Terminal-Bench has introduced a new benchmark category — Challenges — that sits above its existing task suite in both cost and difficulty. The three opening challenges are designed to require days of agent runtime and thousands of dollars in compute. The leaderboards opened this week. No model has scored on any of them.
What the Challenges Are
The three initial challenges expose a tier of software engineering work that current agents have not demonstrably cleared:
Rust Compiler Speedup — Improve rustc compile times on a 52-crate subset of the official rustc-perf benchmark while preserving full correctness. The task requires structural changes: addressing parallelism limits, linking overhead, and codegen bloat. Anthropic ran Claude Opus 4.8 on this task for 12 hours. The agent failed to produce net improvement; its optimization attempts caused regressions.
Inference Engine Code Golf — Write a complete inference engine for Kimi K2.5, implemented in a single C/CUDA file under 25KB. Scored on a Pareto frontier across TTFT, tokens per second, and TPOT. Extends the gpt2-codegolf task from Terminal-Bench 2.0 to a production-scale frontier model.
WASM Render — Implement a pure JavaScript/WebAssembly software renderer supporting WebGL 1.0 and 2.0.
No time or resource limits apply. Outputs are graded on task-specific metrics covering correctness and performance. Each challenge includes a concise project description and an extensive test bed, both provided to the agent at start.
The Benchmark Stack
Terminal-Bench has now stratified the evaluation space into three distinct tiers:
| Benchmark | Task type | Cost per task | Time per task | Solution LOC |
|---|---|---|---|---|
| SWE-bench | Patch a single issue | ~$1–$10 | Minutes to hours | Up to 500 |
| Terminal-Bench 2.0 | Write and execute programs | ~$1–$100 | Minutes to hours | Up to 10,000 |
| Challenges | Build an entire codebase | ~$1,000+ | Days | 10,000–500,000 |
SWE-bench tests whether a model can close a GitHub issue. Terminal-Bench 2.0 tests whether it can implement small programs end-to-end. Challenges test whether it can deliver what a team of senior engineers would call a multi-month project — Rust compiler internals, a production inference engine, a graphics subsystem.
What the Failure Means
Claude Opus 4.8 scored 88.6% on SWE-bench Verified and leads Agent Arena at 8.85% composite. On the Rust compiler task — a single, clearly specified goal with a rigorous test suite — it ran for 12 hours and produced negative results.
The failure is not a benchmark anomaly. Rust’s compiler is a mature codebase with well-understood bottlenecks. The community has already harvested many incremental gains through parallelism and LLVM tuning. Further improvement requires restructuring core subsystems — the kind of work that demands sustained reasoning about large, interdependent codebases across many context windows.
That constraint appears to be where current agents stall. The benchmark makes the gap measurable.
Key Numbers
- Three challenges released: Rust Compiler Speedup, Inference Engine Code Golf, WASM Render
- Claude Opus 4.8: 12-hour attempt on Rust task, regressions only — no net improvement
- All three leaderboards: no scores, rolling out shortly
- Cost threshold: approximately $1,000+ per serious attempt
- SWE-bench ceiling for Opus 4.8: 88.6% (top of the field)
- Agent Arena rank for Opus 4.8 Thinking: 2 of 19 models (8.85% composite)
The Challenges represent the first formal benchmark tier where frontier models have no published score to defend.