NewsBite analysis · AI, Gaming
OpenAI
How enabling two settings tripled our scores on the ARC-AGI-3 benchmark
OpenAI's GPT-5.6 Sol has achieved a major breakthrough by increasing its score on the ARC-AGI-3 benchmark of 2D puzzle games from 7.8% to three times that by utilizing two critical API settings: retained reasoning and compaction. Previously, the model struggled due to limitations in setting retention that hindered its ability to build on prior knowledge during gameplay. By optimizing these settings, GPT-5.6 Sol now retains and builds on its reasoning across turns, allowing it to perform more efficiently. This not only highlights the capabilities of the model itself but also emphasizes how important API settings and harness design are in AI benchmarking.
ExperimentConfidence: MediumShort term horizon
Published 29 Jul 2026, 23:15 UTC3 min read
Visit original article ↗Lead image — OpenAI
Summary
TL;DR
- GPT-5.6 Sol's score on ARC-AGI-3 benchmark tripled after enabling two settings.
- The settings included 'retained reasoning' and 'compaction' tailored for gameplay.
- GPT-5.6 Sol's previous score was only 7.8%, while GPT-5.5 scored 0.4%.
- These adjustments significantly reduced required output tokens by 6x.
- The model now utilizes past reasoning to improve gameplay efficiency.
Interesting facts
- The ARC-AGI-3 benchmark tests AI agents in grasping unfamiliar games without instructions.
- GPT-5.6 Sol previously struggled due to lost reasoning messages concerning its actions.