← Back to feed

Arena.ai published HarnessTax on September 16, 2026, the result of evaluating 7 frontier models — including Claude Fable 5, GPT-5.6 Luna, GPT-5.6 Sol, and Kimi K3 — across three production coding harnesses: Claude Code, Codex CLI, and Pi. The benchmark suite covered SWE-bench Lite and Terminal-Bench 2.0. Three findings headline the paper.

Harness choice does not move the success needle much

The most counterintuitive result: models do not show substantial performance differences across harnesses. The same model run through Claude Code versus a simpler scaffold achieved similar resolve rates on both SWE-bench Lite and Terminal-Bench 2.0. A simple harness can be competitive with a commercial-grade one.

This contradicts a common assumption that investing in a richer agent framework pays off in raw task completion. For models at the frontier — where internal reasoning capability is high — the harness appears to be largely a transport layer, not a performance multiplier.

The cost gap is where harnesses diverge

Where harnesses do differ is in what they bill per successful task. The gap between the cheapest and most expensive harness on the same model was significant. GPT-5.6 Luna achieved the lowest cost per task on both benchmarks — the combination of its pricing and efficient token usage made it the Pareto-optimal choice for cost-sensitive deployments.

Claude Fable 5 led on raw performance, reaching 97.8% on SWE-bench Lite. But different harnesses extracted that performance at meaningfully different prices.

Open-weight models at the cost frontier

Kimi K3, an open-weight model, placed near the Pareto frontier on SWE-bench Lite alongside GPT-5.6 Sol, and sat just below the frontier on Terminal-Bench 2.0. At open-weight pricing, Kimi K3 offers a competitive cost-performance ratio against proprietary models.

What this means for teams building agents

The practical implication is that teams deploying coding agents should benchmark harnesses primarily on cost, not expected capability lift. If the model is capable enough, the harness is unlikely to change whether a task succeeds or fails — it will change what that success costs.

For most use cases, the expensive commercial harness is not buying extra task success. It is buying tooling, workflow integration, and vendor support. Those may be worth it. The task success rate is not the reason to pay for them.

The HarnessTax paper is published at harnesstax.github.io with a BibTeX citation available for academic use.