ARC-AGI-3 Benchmark Rules Clarified and Official Code Released
Recent evaluation methods for large models on the ARC-AGI-3 benchmark have sparked controversy. In response, ARC-AGI founder François Chollet and official member Greg Kamradt have clarified the evaluation rules and API usage, and open-sourced the official testing codebase to ensure fairness and transparency across model evaluations.
Confirmed
- Custom tools banned: François Chollet explicitly prohibits using custom tools developed specifically for this benchmark or systems containing knowledge of the test format and content.
- Fair testing mechanism: Addressing the controversy over Claude Opus, Greg Kamradt clarified that the official testing used the exact same "sliding window" convention for both Anthropic and OpenAI models.
- Open-source evaluation tool: Greg Kamradt shared the official ARC Prize open-source codebase arc-agi-3-benchmarking, which provides complete environment setup and running guides, supporting configuration of various API keys to evaluate large models on complex abstract reasoning tasks.
- API field clarification: Greg Kamradt corrected misconceptions about the optional reasoning field in the API, emphasizing it is not for extracting the model's internal chain-of-thought (CoT) but only for recording and associating external outputs.
Why it matters
A unified and open evaluation standard helps eliminate suspicions of "cheating" among model providers during benchmarking. Explicitly banning custom tools and publishing test code provides developers with a reproducible evaluation environment, further solidifying ARC-AGI's authority as a benchmark for frontier model intelligence.
2026-07-30 ~ 2026-07-30 · 5 related posts
- Episode 1: Opus 5's High ARC-AGI-3 Score Sparks Cheating and Overfitting Controversy(2026-07-25, 11 posts)
- Episode 2: Deep Dive into Opus 5 Hidden Reasoning and ARC-AGI Score(2026-07-25, 2 posts)
- Episode 3: Anthropic's Benchmark Scores Spark Community Trust Crisis(2026-07-25, 2 posts)
- Episode 4: Gary Marcus Says ARC-AGI Name Is Misleading(2026-07-27, 2 posts)
- Episode 5: Human Baselines Missing in AI Evaluations, Highlighting Human-AI Synergy(2026-07-29, 4 posts)
- Episode 6: Optimized Memory Settings Triple GPT-5.6's Score on ARC-AGI-3(2026-07-30, 27 posts)
- Episode 7: ARC-AGI 3 Evaluation Mechanism Under Fire from Developers(2026-07-30, 9 posts)
- Episode 8: Claude Opus ARC-AGI Score Questioned Over API Flaw(2026-07-30, 2 posts)
- Episode 9: ARC-AGI-3 Benchmark Rules Clarified and Official Code Released(2026-07-30, 5 posts)
Primary sources
- [source] Greg Kamradt Responds to Evaluation Dispute: Same Rolling Window Used for Opus and OpenAI — GregKamradt · 2026-07-30
- [source] ARC-AGI-3 Official Benchmarking Repo Goes Open Source — GregKamradt · 2026-07-30
- ARC-AGI-3 Official Benchmarking Repository Open-Sourced — GregKamradt · 2026-07-30
- ARC-AGI-3 API clarification: 'reasoning' field logs model output, not private CoTs — GregKamradt · 2026-07-30
- [source] ARC-AGI Creator Clarifies Rules: No Custom Harnesses for Benchmark Testing — fchollet · 2026-07-30