NewsBite analysis · AI, Programming
The Decoder
OpenAI claims GPT-5.6 Sol beats Opus 5 on ARC-AGI-3 with its latest API and two additional settings
OpenAI has announced that its GPT-5.6 Sol model achieves a surpassing score of 38.3 percent on ARC-AGI-3, compared to Anthropic's Opus 5 score of 30.2 percent. However, François Chollet, co-founder of the ARC Prize, highlighted potential issues with fairness in the benchmark due to different test setups. He noted that the API settings used by OpenAI, which keep context between steps, could give them an edge in this evaluation. While OpenAI claims these models' configurations warrant their performance metrics, ARC Prize maintains that standardized testing methods are critical for equitable comparisons.
MonitorConfidence: MediumShort term horizon
Published 30 Jul 2026, 10:10 UTC2 min read
Visit original article ↗Lead image — The Decoder
Summary
TL;DR
- OpenAI claims GPT-5.6 Sol scores 38.3% on ARC-AGI-3.
- Anthropic's Opus 5 previously achieved 30.2%.
- Different API settings may create a parity issue.
- Official ARC scores are based on standardized approaches.
- Debate around fairness in benchmark comparisons continues.
Interesting facts
- ARC-AGI-3 aims to evaluate pure model performance rather than setup.
- OpenAI’s customized settings include features not available in the official test environment.
Coverage gaps