Two API Tweaks Boost OpenAI Model's ARC-AGI-3 Score and Slash Compute
xiaohu · x · 2026-07-30
OpenAI recently reviewed why its model underperformed on the ARC-AGI-3 benchmark, identifying the root cause in the evaluation harness: private reasoning was wiped after each step, and early context was deleted when limits were hit, causing the model to lose track of history.
They fixed this by enabling just two API settings:
- Retain Reasoning: Using the Responses API to pass back the previous response ID, automatically preserving intermediate reasoning.
- Context Compaction: Enabling compaction to summarize context when full, rather than dropping the beginning.
This yielded dramatic results: GPT-5.6 Sol's score on the public set jumped from 13.3% to 38.3%. It also massively reduced costs, with output tokens per game dropping from 2.9 million to 490k (about 1/6th) on the highest setting.
Related event: Optimizing Test Harness Triples GPT-5.6 Score on ARC-AGI-3(21 posts)→
More from coding & agent
- Agent Benchmark Harness Criticized: Gimping Long-Term Memory Skews Efficiency Tests — teortaxesTex · 2026-07-30
- Hardcore Metaphor: Extending LLM Context is Like Organ Transplant Surgery — Sauers_ · 2026-07-30
- Study: Agentic Harnesses Significantly Improve LLM Compositional Generalization — lateinteraction · 2026-07-30
- Real Costs of AI Coding Agent: Median PR Cost is $27.47 — zeeg · 2026-07-30
- Hands-on with Codex Micro: A Core Component for Agent Workflows — rudrank · 2026-07-30
- Grok Build Update: Enhanced Headless Streaming and Tool Call Support — XFreeze · 2026-07-30