Two API Settings Tripled OpenAI Model's Score on ARC-AGI-3
机器之心 · wechat · 2026-07-30
An OpenAI blog post revealed how the evaluation environment drastically impacts AI Agent performance. Under the official generic harness of the ARC-AGI-3 benchmark, GPT-5.6 Sol scored only 13.3%. However, switching to a setup closer to ChatGPT and Codex boosted the score to 38.3%—a threefold increase—while reducing token consumption by 6x.
OpenAI identified two mechanisms in the original harness that severely limited the model's long-term memory:
- Discarded Reasoning: The model's private reasoning was deleted after every action, forcing it to re-analyze the current state from scratch.
- Truncated Context: A rolling cutoff deleted the oldest messages, causing the model to lose early critical actions and observations.
To fix this, OpenAI enabled two key API settings:
- Retained Reasoning: Keeping internal reasoning as context allows the model to act coherently based on past understanding without re-thinking every step.
- Context Compaction: Replacing crude truncation with summarization preserves vital historical information while managing context size.
This experiment proves that as Agents tackle longer, complex tasks, benchmarks are evaluating the entire runtime environment, not just the model itself.
Related event: Optimized Memory Settings Triple GPT-5.6's Score on ARC-AGI-3(27 posts)→
More from coding & agent
- 10 open-source repos that let your AI agent reverse-engineer apps and games — victor_explore · 2026-10-16
- The new CTO job: agent management and QA loops, 'cable management' instead of coding — HankYeomans · 2026-10-16
- Why company AI agents go stale: adoption needs in-house knowledge, not just better agents — EdenEmarco177 · 2026-10-16
- Open-source MCP lets coding agents search Obsidian's 9,000-plugin directory before reinventing wheels — pitfa19 · 2026-10-16
- Cheng Lou open-sources Handwired: hand-written weights turn code into training-free neural nets — dotey · 2026-10-16
- Building a portable context vault in Notion so every AI agent knows you from day one — ivanhzhao · 2026-10-16