← Back to feed

GPT-5.6 Sol scored 7.8% on ARC-AGI-3 with the official harness. On the public task subset, the official number is 13.3%. OpenAI published a blog post on July 29 showing that enabling two API settings pushed that same model to 38.3% on the same public task set — a 25-point swing from a configuration choice, not a model change.

Claude Opus 5 holds the official ARC-AGI-3 record at 30.2%, covered here when it launched in late July as the best score by 4x over the previous leader. OpenAI’s production-harness result now exceeds it. Neither score is directly comparable: ARC-AGI-3 uses Relative Human Action Efficiency (RHAE), which scores model performance relative to a human baseline estimated at around 48%.

The Two Settings

The official ARC-AGI-3 harness is designed to be generic and model-agnostic. It does two things that turned out to matter a great deal.

First, it discards all private reasoning after each action. GPT-5.6 Sol, like all modern reasoning models, generates hidden thinking tokens before every response. Those thoughts contain plans, inferences about game rules, and working hypotheses about what to try next. The official harness throws them away. On the next action, the model must reconstruct its understanding of the game from scratch using only the visible action log.

Second, the official harness applies rolling truncation: as game history grows, the oldest actions fall off the visible context window. The model loses memory of its past moves as well as its past reasoning.

OpenAI implemented the same benchmark tasks using their Responses API, which retains reasoning across tool calls by default and replaces rolling truncation with compaction (context summarisation). These are not experimental settings — they are how GPT-5.6 Sol runs in ChatGPT and Codex. The blog post authors are Ilan Bigio and Ted Sanders.

What Changed in Practice

With retained reasoning, the model spent less time before each action. It no longer had to re-infer game rules from a sparse action history on every turn. With compaction, it retained awareness of earlier game states even in long sessions.

On one specific game in the leaderboard, the official results show no frontier model solving any level beyond the first. With the production harness, GPT-5.6 Sol solved all six levels.

The Benchmark Methodology Argument

ARC’s stated rationale for the generic harness was fairness: a simple setup removes developer-specific optimisations and makes model-to-model comparisons cleaner. OpenAI’s results challenge this directly. Their argument is that a harness designed to be neutral is actually penalising reasoning models in a way that does not reflect their real-world capability, because reasoning retention is now a standard production feature, not a specialist optimisation.

This is not the first time harness design has driven benchmark headlines. SWE-bench Verified scores diverge substantially based on scaffold, with multi-rollout systems regularly outperforming bash-only harnesses on the same underlying models. The ARC-AGI-3 case is starker because the same model, same tasks, and a single config change produce a 25-point RHAE difference.

The broader implication: benchmark scores now carry a hidden parameter. The question is not just “what model?” but “what harness, and does that harness match how the model was trained to operate?”

What This Means for the Leaderboard

The 38.3% result is not submitted to the official ARC-AGI-3 leaderboard. It does not update the rankings. Claude Opus 5 at 30.2% remains the official SOTA.

ARC has not responded publicly as of publication. The question for evaluation methodology is whether the official harness should adopt reasoning retention as a standard option, or whether the intentional penalty for non-standard features is the point.

Human average on ARC-AGI-3 sits around 48% based on official gameplay logs. The gap between human and frontier — even at 38.3% — remains substantial.