GPT-5.6 Sol Triples on ARC-AGI-3 Without a Weight Change — Two API Settings Did It
OpenAI published a research post on July 29 showing that GPT-5.6 Sol scored 38.3% on the ARC-AGI-3 public task set — nearly triple its 13.3% baseline — without any change to the model’s weights. Two settings in the Responses API made the difference: retained reasoning and compaction.
On the official ARC Prize leaderboard, Sol’s verified score is 7.8%. Claude Opus 5, which holds the current ARC-AGI-3 record, sits at 30.2%. The 38.3% and 30.2% numbers come from different task sets under different harnesses. Multiple outlets ran the headline “Sol beats Opus 5 on ARC-AGI-3.” That comparison is not supported by the data. OpenAI’s own post does not mention Opus 5 at all — the framing came from an employee post on X and then from downstream coverage.
What the Standard Harness Was Doing
The official ARC-AGI-3 harness discarded the model’s private reasoning after each game action. Sol could see a log of its past moves, but not the plans, hypotheses, or insights behind them. Every turn started by reconstructing the puzzle from the move history alone. On top of that, once the conversation exceeded 175,000 characters, the harness deleted the oldest messages outright.
For a benchmark whose entire premise is learning an unfamiliar environment over time, those two defaults produce something close to a worst case for any model that builds understanding iteratively.
OpenAI changed both:
- Retained reasoning: The model’s chain of thought is preserved across turns rather than discarded after each action.
- Compaction: Long conversations are summarized rather than truncated, so observations from early game states survive into later turns.
Both are existing settings in the Responses API — the same infrastructure that runs ChatGPT and Codex. Neither was purpose-built for ARC-AGI-3.
The Performance Jump
| Harness | Task set | GPT-5.6 Sol | Claude Opus 5 |
|---|---|---|---|
| OpenAI Responses API (retained + compaction) | Public | 38.3% | — |
| ARC Prize standard protocol | Public (max effort) | 13.3% | ~30.2% |
| ARC Prize standard protocol | Semi-private | 7.8% | ~30.2% |
| Human testers | Public | 48% | 48% |
Output tokens fell roughly sixfold while the score tripled. Higher and cheaper simultaneously is the signal that something real changed, not just a metric inflation: when Sol could see its earlier reasoning, it stopped deriving the game from scratch before every move, which cut output while improving strategy coherence.
ARC Prize co-founder François Chollet acknowledged that general-purpose API settings not built specifically for ARC-AGI-3 are acceptable for reporting alongside official results. He noted the different settings create “a potential parity issue” but did not change the official evaluation protocol.
The Harness Design Implication
The result’s significance is not the comparative claim — that was wrong when it ran. It is the demonstration of how much harness design moves the needle on a benchmark that was supposed to measure generalization.
ARC Prize has previously documented custom harnesses taking a model from zero to over 97% on one environment while doing nothing on a different one. The same dynamics apply here in the opposite direction: a default harness that discards reasoning after every turn and truncates history aggressively is not a neutral measurement instrument for multi-step strategy tasks.
For developers building agentic systems, the practical implication is direct. Whether your model retains its reasoning between steps, and how you handle context overflow, are decisions that affect task performance as much as which model you select. The settings you never explicitly configured are part of your benchmark score — and part of your production system’s behavior.
The 7.8% vs 38.3% gap also does something useful: it shows exactly what the standardized harness costs this particular model on this particular benchmark type. That is a real number with real operational meaning, not a confound to be dismissed.
Key Numbers
- 13.3% to 38.3%: Sol’s gain on ARC-AGI-3 public task set under OpenAI’s own harness
- 7.8%: Sol’s verified score on the semi-private set under ARC Prize standard protocol
- 30.2%: Opus 5’s verified ARC-AGI-3 record under ARC Prize protocol
- 48%: Average human tester score on the same benchmark
- 6x: Reduction in output tokens when retained reasoning + compaction is enabled
- July 29, 2026: Date of OpenAI’s post by Ilan Bigio and Ted Sanders