OpenAI Triples GPT-5.6 Score on ARC-AGI-3 by Tweaking API Settings
GregKamradt · x · 2026-07-30
OpenAI's internal testing revealed that enabling specific API settings, specifically provider-managed conversation state, tripled the GPT-5.6 Sol's score on long-horizon tasks like ARC-AGI-3 while using 6x fewer output tokens.
Responding to this, ARC Prize acknowledged that such "harness" approaches improve cross-turn continuity and performance. However, to ensure fair comparison across providers, ARC's official verified scores will continue to use a "no harness" standard. All systems receive identical observations, prompts, and action limits, with conversation state managed entirely client-side to prevent format-specific targeting. ARC is actively working with OpenAI and other labs to figure out how to best incorporate these server-side capabilities.
Related event: Optimized Memory Settings Triple GPT-5.6's Score on ARC-AGI-3(27 posts)→
More from Models
- Opus 5.5 still leads a month later, but open-weight decision models are closing in — airesearch12 · 2026-10-16
- Google restricts Gemini Pro access for Google Plus subscribers, matching free tier — EmergencyLess8861 · 2026-10-16
- Meta releases Muse Spark Open, trained with vLLM-powered RL inference from day one — colinmcnamara · 2026-10-16
- Meta open-sources Muse Spark: runs on two Mac Studios, license bans passing output off as human-written — mawcopolow · 2026-10-16
- Golden Gate Claude explainer resurfaces: interpretability premise still holds — HarperSCarroll · 2026-10-16
- Hugging Face livestreams release of a new open-weight model — huggingface · 2026-10-16