CL-SN55 941 —
GPT-61S 890 +8.8%
GPT-56T 861 —
GPT-6S 847 +1.2%
GPT-56SC 847 +1.2%
CL-OP55X 843 +1.4%
GROK-46H 843 +1.4%
GLM-52 843 +1.2%
DSK-V4PH 843 +1.4%
GLM-5 826 +10.3%
MUSE-SPK 804 —
QWEN-38X 795 +10.1%
GPT-6A 782 +11.1%
CL-FAB5H 761 +1.6%
CL-OP5H 736 +1.7%
CL-OP5X 734 +1.7%
GEM-37FH 732 +1.7%
KIMI-K3X 719 +0.1%
CL-OP46H 719 +1.7%
CL-OP55H 717 +1.7%
CL-OP47H 708 +1.7%
GEM-38FH 689 +0.3%
GPT-56S 581 -0.5%
CL-OP47 546 -0.4%
INKL 531 —
GEM-3P 495 —
CL-OP46 491 —
GEM-31P 487 -0.2%
CL-OP48 484 —
GPT-6L 436 —
CL-SN55 941 —
GPT-61S 890 +8.8%
GPT-56T 861 —
GPT-6S 847 +1.2%
GPT-56SC 847 +1.2%
CL-OP55X 843 +1.4%
GROK-46H 843 +1.4%
GLM-52 843 +1.2%
DSK-V4PH 843 +1.4%
GLM-5 826 +10.3%
MUSE-SPK 804 —
QWEN-38X 795 +10.1%
GPT-6A 782 +11.1%
CL-FAB5H 761 +1.6%
CL-OP5H 736 +1.7%
CL-OP5X 734 +1.7%
GEM-37FH 732 +1.7%
KIMI-K3X 719 +0.1%
CL-OP46H 719 +1.7%
CL-OP55H 717 +1.7%
CL-OP47H 708 +1.7%
GEM-38FH 689 +0.3%
GPT-56S 581 -0.5%
CL-OP47 546 -0.4%
INKL 531 —
GEM-3P 495 —
CL-OP46 491 —
GEM-31P 487 -0.2%
CL-OP48 484 —
GPT-6L 436 —
← Back to feed

Terminal-Bench 4.0: 8 Tasks Pruned for Saturation, 8-Hour Flat Timeout, Sonnet 5 Flagged

Terminal-Bench 4.0 is a maintenance release, not a new benchmark. It exists because benchmarks are complex simulations that contain bugs and need ongoing upkeep — and because a task that every frontier model solves 5/5 no longer tells you anything.

Eight tasks were removed. Nineteen were fixed. All tasks now run against a flat 8-hour agent timeout. And Claude Sonnet 5 got flagged for a specific resource problem that other models don’t share.

What Changed

Removed tasks (8):

  • 2 saturated: every model in the current frontier generation solves them 5/5. No further discriminative value.
  • 2 refusals: agents consistently refuse the task, making results noise rather than signal.
  • 2 public solutions: the tasks have leaked; results reflect memorization, not capability.
  • 2 quality/compatibility: unresolved issues that couldn’t be repaired.

Fixed tasks (19): Instructions, environments, and verifiers updated to address flakiness and misspecification found in leaderboard runs and user reports.

Timeout calibration: All tasks now set to a flat 8-hour agent timeout. The methodology follows Anthropic’s “Quantifying infrastructure noise in agentic coding evals” — run tasks with large resource headroom first, then select a setting that doesn’t affect final metrics. Frontier models now rarely encounter timeouts.

Sonnet 5’s Token Problem

TB 4.0 surfaced a specific issue with Claude Sonnet 5: the model uses 21.6B tokens on its leaderboard run versus 6.5B for Opus 5. It also hit timeouts and output-token-exceeded errors. The 128k max-output-tokens setting was not enabled, which may have contributed.

This is a token efficiency gap, not a capability gap — but it has practical implications for anyone running Sonnet 5 in long-horizon agentic workflows where token budget matters.

Semantic Versioning

TB 4.0 instead of 3.1 because Terminal-Bench now uses semantic versioning. Resource calibration changes and task set changes are breaking — they require re-running trials, so they warrant a major version bump.

The next milestones are already public on GitHub:

  • TB 4.1: tamper-resistant verifiers
  • TB 5.0: new tasks

The benchmark is built on the Harbor framework and is available at hub.harborframework.com/datasets/terminal-bench/terminal-bench/4.

harbor run -d terminal-bench/terminal-bench@4.0.0

Leaderboard grants are supported by OpenAI, Anthropic, Z.ai, SpaceX AI, and the Laude Institute.