Optimized Memory Settings Triple GPT-5.6's Score on ARC-AGI-3
OpenAI officially confirmed that by enabling "reasoning retention" and "context compaction" in the Responses API, GPT-5.6 Sol's score on the ARC-AGI-3 public test set surged from 13.3% to 38.3% (a 188% increase), reaching SOTA levels while reducing token consumption to 1/6th of the original. This reveals that the memory management mechanism within the test harness, rather than pure model reasoning ability, is the key determinant of performance in complex long-horizon tasks.
Confirmed
- GPT-5.6 Sol initially performed poorly on the ARC-AGI-3 benchmark, an issue first highlighted in tests by developer @sandersted.
- By enabling two specific API settings used internally by ChatGPT and Codex, the model's score increased roughly threefold (from 13.3% to 38.3%), alongside a sixfold improvement in token efficiency (reducing consumption to 1/6th).
- The two key settings are reasoning retention and context compaction, which allow the model to remember previous thought processes and operate continuously across multiple context windows.
- OpenAI engineer @ilanbigio noted that standard test harnesses discard the model's reasoning process after each step and lose early operation records when the context fills up, limiting the model's evaluated performance in isolated environments.
Why it matters
- This discovery challenges the current industry-standard isolated evaluation methods. Developers like @Angaisb pointed out that complex tasks can be easily solved by optimizing the test harness, meaning that merely comparing raw benchmark scores does not truly represent the actual intelligence of different AI systems.
- The deep synergy between product frameworks (like ChatGPT and Codex's internal mechanisms) and model capabilities has a decisive impact on complex task performance, indicating that the performance bottleneck often lies in context management rather than the model itself.
2026-07-30 ~ 2026-08-01 · 27 related posts
- Episode 1: Opus 5's High ARC-AGI-3 Score Sparks Cheating and Overfitting Controversy(2026-07-25, 11 posts)
- Episode 2: Deep Dive into Opus 5 Hidden Reasoning and ARC-AGI Score(2026-07-25, 2 posts)
- Episode 3: Anthropic's Benchmark Scores Spark Community Trust Crisis(2026-07-25, 2 posts)
- Episode 4: Gary Marcus Says ARC-AGI Name Is Misleading(2026-07-27, 2 posts)
- Episode 5: Human Baselines Missing in AI Evaluations, Highlighting Human-AI Synergy(2026-07-29, 4 posts)
- Episode 6: Optimized Memory Settings Triple GPT-5.6's Score on ARC-AGI-3(2026-07-30, 27 posts)
- Episode 7: ARC-AGI 3 Evaluation Mechanism Under Fire from Developers(2026-07-30, 9 posts)
- Episode 8: Claude Opus ARC-AGI Score Questioned Over API Flaw(2026-07-30, 2 posts)
- Episode 9: ARC-AGI-3 Benchmark Rules Clarified and Official Code Released(2026-07-30, 5 posts)
Primary sources
- GPT-5.6 ARC-AGI-3 scores surge 3x with specific API settings enabled — sandersted · 2026-07-30
- GPT-5.6 ARC-AGI-3 Score Jumps 3x With Specific API Settings — sandersted · 2026-07-30
- GPT-5.6 ARC-AGI-3 Score Triples When Enabling Thought Memory — sandersted · 2026-07-30
- [source] OpenAI Reveals Compaction Boosts GPT-5.6 ARC-AGI-3 Score to 38% — ilanbigio · 2026-07-30
- Optimized Harness Achieves SoTA on ARC-AGI-3, Questioning Benchmark Validity — Angaisb_ · 2026-07-30
- OpenAI Tripled Its ARC-AGI-3 Scores by Enabling Just Two Settings — ObiWanCanownme · 2026-07-30
- [source] OpenAI: Optimized API Settings Triple GPT-5.6 Sol's Score on ARC-AGI-3 — OpenAI · 2026-07-30
- GPT-5.6 Sol Achieves SOTA on ARC-AGI-3 via Context Compaction — daniel_mac8 · 2026-07-30
- Retaining Reasoning Across Turns Boosts GPT-5.6 Sol on ARC-AGI-3 by ~3x — charliermarsh · 2026-07-30
- GPT-5.6 Sol Reasoning Details: Lack of Memory Forces Re-learning Every Step — charliermarsh · 2026-07-30
- Harness Matters: GPT-5.6 Hits SoTA on ARC-AGI-3 with Two Setting Tweaks — TheZachMueller · 2026-07-30
- GPT Performance Jumps 188% with 6x Fewer Tokens via Context Compaction — soumitrashukla9 · 2026-07-30
- Two API Settings Triple GPT-5.6's ARC-AGI-3 Score While Cutting Tokens — gabrielchua · 2026-07-30
- OpenAI's New API Harness Boosts GPT-5.6 Score by 188% Using 6x Fewer Tokens — arthurcolle · 2026-07-30
- Optimizing Context Management Triples Score on ARC-AGI-3 — ns123abc · 2026-07-30
- OpenAI Test: Optimized Harness Boosts Model Scores by 188% with 6x Fewer Tokens — ChrisUniverse · 2026-07-31
- OpenAI Reveals GPT-5.6 Eval: Tweaking Memory Settings Tripled Scores with 6x Fewer Tokens — TheTuringPost · 2026-08-01
10 near-duplicate retellings: sandersted · ilanbigio · OpenAI · tw_killian · soumitrashukla9 · GregKamradt · xiaohu · 机器之心 · rohanpaul_ai · KeanuRave100