CL-SN55 941 —
GPT-61S 885 —
GPT-56T 861 —
GPT-6S 825 —
CL-OP55X 818 -0.1%
MUSE-SPK 793 —
QWEN-38X 779 —
GPT-6A 778 —
CL-OP5H 711 -0.1%
CL-OP5X 709 -0.1%
KIMI-K3X 699 -0.1%
CL-OP46H 694 -0.1%
CL-FAB5H 692 -0.1%
CL-OP55H 692 -0.1%
CL-OP47H 683 -0.1%
GEM-37FH 648 -0.2%
GEM-38FH 633 -0.2%
INKL 531 —
CL-OP47 529 -0.2%
GPT-56S 529 -5.9%
GEM-3P 493 —
CL-OP46 488 —
CL-OP48 481 —
GEM-31P 453 -0.2%
GPT-6L 436 —
CL-SN55 941 —
GPT-61S 885 —
GPT-56T 861 —
GPT-6S 825 —
CL-OP55X 818 -0.1%
MUSE-SPK 793 —
QWEN-38X 779 —
GPT-6A 778 —
CL-OP5H 711 -0.1%
CL-OP5X 709 -0.1%
KIMI-K3X 699 -0.1%
CL-OP46H 694 -0.1%
CL-FAB5H 692 -0.1%
CL-OP55H 692 -0.1%
CL-OP47H 683 -0.1%
GEM-37FH 648 -0.2%
GEM-38FH 633 -0.2%
INKL 531 —
CL-OP47 529 -0.2%
GPT-56S 529 -5.9%
GEM-3P 493 —
CL-OP46 488 —
CL-OP48 481 —
GEM-31P 453 -0.2%
GPT-6L 436 —
← Back to feed

Terminal-Bench 4.0: 8 Tasks Pruned for Saturation, 8-Hour Flat Timeout, Sonnet 5 Flagged

Terminal-Bench 4.0 is a maintenance release, not a new benchmark. It exists because benchmarks are complex simulations that contain bugs and need ongoing upkeep — and because a task that every frontier model solves 5/5 no longer tells you anything.

Eight tasks were removed. Nineteen were fixed. All tasks now run against a flat 8-hour agent timeout. And Claude Sonnet 5 got flagged for a specific resource problem that other models don’t share.

What Changed

Removed tasks (8):

  • 2 saturated: every model in the current frontier generation solves them 5/5. No further discriminative value.
  • 2 refusals: agents consistently refuse the task, making results noise rather than signal.
  • 2 public solutions: the tasks have leaked; results reflect memorization, not capability.
  • 2 quality/compatibility: unresolved issues that couldn’t be repaired.

Fixed tasks (19): Instructions, environments, and verifiers updated to address flakiness and misspecification found in leaderboard runs and user reports.

Timeout calibration: All tasks now set to a flat 8-hour agent timeout. The methodology follows Anthropic’s “Quantifying infrastructure noise in agentic coding evals” — run tasks with large resource headroom first, then select a setting that doesn’t affect final metrics. Frontier models now rarely encounter timeouts.

Sonnet 5’s Token Problem

TB 4.0 surfaced a specific issue with Claude Sonnet 5: the model uses 21.6B tokens on its leaderboard run versus 6.5B for Opus 5. It also hit timeouts and output-token-exceeded errors. The 128k max-output-tokens setting was not enabled, which may have contributed.

This is a token efficiency gap, not a capability gap — but it has practical implications for anyone running Sonnet 5 in long-horizon agentic workflows where token budget matters.

Semantic Versioning

TB 4.0 instead of 3.1 because Terminal-Bench now uses semantic versioning. Resource calibration changes and task set changes are breaking — they require re-running trials, so they warrant a major version bump.

The next milestones are already public on GitHub:

  • TB 4.1: tamper-resistant verifiers
  • TB 5.0: new tasks

The benchmark is built on the Harbor framework and is available at hub.harborframework.com/datasets/terminal-bench/terminal-bench/4.

harbor run -d terminal-bench/terminal-bench@4.0.0

Leaderboard grants are supported by OpenAI, Anthropic, Z.ai, SpaceX AI, and the Laude Institute.