Two API Tweaks Boost OpenAI Model's ARC-AGI-3 Score and Slash Compute
xiaohu · x · 2026-07-30
OpenAI recently reviewed why its model underperformed on the ARC-AGI-3 benchmark, identifying the root cause in the evaluation harness: private reasoning was wiped after each step, and early context was deleted when limits were hit, causing the model to lose track of history.
They fixed this by enabling just two API settings:
- Retain Reasoning: Using the Responses API to pass back the previous response ID, automatically preserving intermediate reasoning.
- Context Compaction: Enabling compaction to summarize context when full, rather than dropping the beginning.
This yielded dramatic results: GPT-5.6 Sol's score on the public set jumped from 13.3% to 38.3%. It also massively reduced costs, with output tokens per game dropping from 2.9 million to 490k (about 1/6th) on the highest setting.
Related event: Optimized Memory Settings Triple GPT-5.6's Score on ARC-AGI-3(27 posts)→
More from coding & agent
- Gemini shown off modding Minecraft to turn players into any Spider-Man — zacharynado · 2026-10-16
- IBM engineer maps 5 levels of AI coding, from manual to a 3-day agent software factory — TejasKumar_ · 2026-10-16
- 10 open-source repos that let your AI agent reverse-engineer apps and games — victor_explore · 2026-10-16
- The new CTO job: agent management and QA loops, 'cable management' instead of coding — HankYeomans · 2026-10-16
- Why company AI agents go stale: adoption needs in-house knowledge, not just better agents — EdenEmarco177 · 2026-10-16
- Open-source MCP lets coding agents search Obsidian's 9,000-plugin directory before reinventing wheels — pitfa19 · 2026-10-16