ARC-AGI Creator Clarifies Rules: No Custom Harnesses for Benchmark Testing
fchollet · x · 2026-07-30
François Chollet has clarified the guidelines for using harnesses when evaluating models on ARC-AGI-3.
- Not okay: Harnesses custom-built to solve the benchmark or those containing specific knowledge about the benchmark's format or contents.
- Fine: General-purpose API settings that are available to all users and were not developed specifically for ARC-AGI-3.
He noted that there has been a lot of back-and-forth with OpenAI regarding optimal testing methods, particularly concerning context compaction, and is pleased they are figuring it out. While different providers using varying settings introduces a potential parity issue, Chollet believes it is acceptable as long as the settings and costs are transparently reported.
Related event: ARC-AGI-3 Benchmark Rules Clarified and Official Code Released(5 posts)→
More from Models
- First Impressions: Gemini 4 Argon 'Incredible Writer' Beats Opus 5.5 and GPT 6 Astra — soumitrashukla9 · 2026-10-16
- Trying Grok bot surfaces an unpaid Cursor balance, blocking account merging — haydendevs · 2026-10-16
- TensorFold 1.0.12 runs 120B vision model on 1-2 DGX Sparks, adds Mac image input — HankYeomans · 2026-10-16
- Leaked screenshot shows unannounced Gemini 4 Argon gated to Ultra and paid Pro subscribers — il_-bro · 2026-10-16
- Google: Gemini 4 Argon lands in Antigravity, code submissions up 35%, rollbacks down ~33% — tekbog · 2026-10-16
- Google's "ungrounded beliefs" classifier injections derail Opus 4.1's reasoning chain — liminal_bardo · 2026-10-16