NewsBite analysis · AI, Data
The Decoder
OpenAI claims GPT-5.6 Sol beats Opus 5 on ARC-AGI-3 but only with its own custom test harness
OpenAI recently announced that its GPT-5.6 Sol model outperformed Anthropic's Opus 5 on the ARC-AGI-3 logic benchmark with a score of 38.3%. However, this achievement was realized using a custom test configuration, which involved modifications to its Responses API. Under these conditions, Opus 5 scored 30.2%. In contrast, when using the official test setup, GPT-5.6 Sol only achieved 7.8%, highlighting the impact of the surrounding technical framework on benchmark results. This scenario raises questions about the validity of comparisons when the testing methods differ significantly.
MonitorConfidence: MediumShort term horizon
Published 30 Jul 2026, 08:10 UTC1 min read
Visit original article ↗Lead image — The Decoder
Summary
TL;DR
- OpenAI claims GPT-5.6 Sol beats Opus 5 on ARC-AGI-3.
- GPT-5.6 Sol scored 38.3% using a custom test harness.
- Opus 5 achieved a score of 30.2% under the same benchmark.
- Standard testing showed GPT-5.6 Sol at only 7.8%.
- OpenAI emphasizes that benchmarks involve testing conditions.
Interesting facts
- The custom test utilized features like 'Retained Reasoning' and 'Compaction'.
- Official testing protocols are designed to evaluate pure model performance, not configurations.
- OpenAI argues that the technical setup can influence model scores, which complicates direct comparisons.