Harbor-Index 1.0: 82 Tasks, 54 Benchmarks, No Frontier Agent Clears 30%
The team behind Terminal-Bench has published Harbor-Index 1.0, a compact meta-benchmark built to answer a specific question: which 82 tasks from the entire landscape of agentic evaluation are both genuinely hard and definitely working?
The answer required screening 6,627 candidates across 54 benchmarks, spending $300K in compute and 226B tokens, and running 14 human reviewers plus a senior panel through an audit-and-fix loop. The payoff is a suite that costs a frontier agent a few hundred dollars to run end-to-end.
No agent-model pair clears 30%. GPT-5.5 on the Codex CLI leads at 25.6%.
What Gets Cut and Why
The hard insight driving Harbor-Index is that most zero-pass tasks aren’t hard — they’re broken. A task that no agent ever solves might have a verifier that disagrees with the instruction, a success condition that’s underspecified, or an environment missing a file the task needs. Distinguishing genuine difficulty from defective tasks is most of what went into the curation process.
The funnel was three stages:
| Stage | Criterion | Tasks remaining |
|---|---|---|
| Candidate pool | 54 benchmarks via Harbor adapters | 6,627 |
| Difficulty filter | Frontier mix solves ≤33% of trials | 1,311 |
| AI audit | Gemini 3 Flash flags broken tasks | 307 |
| Human audit | 14 reviewers + senior panel | 100 |
| Audit-and-fix loop | Repaired, re-run, unsalvageable dropped | 82 |
The 82 final tasks span 29 of the original 54 benchmarks. The largest contributors to the final set are HLE (8 tasks), GSO (7), ARC-AGI-2 (5), AlgoTune (5), BIX-Bench (5), GAIA2 (5), and SWE-bench Verified (5).
The Leaderboard
Results are plotted as pass rate versus cost per run, with cost reconstructed from token usage at public API rates. Open models routed through Claude Code on OpenRouter see inflated costs from low cache-hit rates, pushing them off the Pareto frontier compared to the same models on Terminus-2.
GPT-5.5 on Codex CLI sits at the top at 25.6%. The dashed Pareto line separates models with good cost-performance ratios from those paying more for similar or worse results.
No model tested breaks 30%, including closed frontier models from OpenAI, Anthropic, and Google.
Why This Matters
Agentic benchmarks have been multiplying fast. Evaluating a new agent across the full landscape — software engineering, scientific reasoning, search, cybersecurity, long-horizon tool use — is expensive in time and compute. Harbor-Index compresses that into 82 tasks that stay genuinely hard while enabling reliable cross-agent comparison.
The Harbor adapter framework is what makes this portable. Adapters translate heterogeneous benchmarks into a single standard format so any agent and model can run through one inference and evaluation pipeline. Parity experiments verify that a task run through Harbor matches the original benchmark’s behavior and evaluation semantics.
Harbor-Index 1.0 is available on Harbor Hub at hub.harborframework.com/datasets/harbor-index/harbor-index-1.0. The adapter framework is open on GitHub under the harbor-framework org.