ARC-AGI 3 Evaluation Mechanism Under Fire from Developers

Recent developers have strongly questioned the fairness and evaluation mechanism of the ARC-AGI 3 benchmark, arguing that the current test framework design, scoring rules, and even testing motives are severely distorted and cannot truly measure general AI capabilities.

Confirmed

• Framework limits memory: Developers @Glittering-Neck-2505 and @teortaxesTex point out that the current unified test framework deliberately prevents reasoning agents from maintaining context across operations, forcing models to constantly "forget." This practice, aimed at cross-provider standardization (using the same completion-style endpoint and not passing back reasoning logs), limits agent long-term memory and fails to effectively measure true model efficiency.
• Evaluation framework significantly affects scores: Developers @ImaginaryDinner2710 and @ilanbigio (via @dkundel) note that model scores heavily depend on the test harness. For ARC-AGI, an OpenAI model scored only 13% due to improper invocation, jumping to 38% after correction. Additionally, just two adjustments to the harness settings led to a significant improvement for Claude 3.5 Sonnet. Properly tracking reasoning and using context compression are crucial for achieving true high scores.
• Scoring mechanism criticized: Developer @mgostIH found that current harnesses have saturated ARC-AGI-3; a small amount of reinforcement learning (RL) can greatly boost scores. He also strongly criticizes the benchmark's use of squaring the percentage of solved problems, calling it baseless and "myth-making."
• Alleged malicious tuning: Developer @iruletheworldmo harshly criticizes ARC-AGI as a catalyst for public attention, suggesting the testers may have maliciously lowered model scores to create the illusion that the benchmark is extremely hard to saturate.
• Unfair base model testing: ML Street Talk (via @burnytech) argues that modern AI is essentially a hybrid neuro-symbolic system, and a single-context reasoning window is insufficient for adaptation, so directly testing base models on ARC-AGI3 is unfair.

Unconfirmed

• Whether testers subjectively "maliciously" tuned parameters to create the illusion of extreme difficulty (currently only a harsh accusation by @iruletheworldmo).
• Whether some models used a different harness in internal preview tests than in public evaluations (question raised by @LiangSong850509).
• Whether the industry can reach consensus on establishing and adopting a unified agent framework for testing different models (currently only a suggestion from ML Street Talk).

Why it matters

• Affects benchmark credibility: If the test mechanism can be easily gamed with a small amount of RL and saturates, the scoring formula is controversial, and there is even potential for artificially lowering scores, the benchmark's authority will be greatly diminished.
• Evaluation must reflect real usage: Developers emphasize that in real development scenarios, models always run with specific agent frameworks. When designing cross-platform standardized tests, one should not sacrifice core agent capabilities (like long-term memory). The industry urgently needs to abandon "bare" benchmarks.

2026-07-30 ~ 2026-08-01 · 9 related posts

Full story(9 episodes)→

• Episode 1: Opus 5's High ARC-AGI-3 Score Sparks Cheating and Overfitting Controversy(2026-07-25, 11 posts)
• Episode 2: Deep Dive into Opus 5 Hidden Reasoning and ARC-AGI Score(2026-07-25, 2 posts)
• Episode 3: Anthropic's Benchmark Scores Spark Community Trust Crisis(2026-07-25, 2 posts)
• Episode 4: Gary Marcus Says ARC-AGI Name Is Misleading(2026-07-27, 2 posts)
• Episode 5: Human Baselines Missing in AI Evaluations, Highlighting Human-AI Synergy(2026-07-29, 4 posts)
• Episode 6: Optimized Memory Settings Triple GPT-5.6's Score on ARC-AGI-3(2026-07-30, 27 posts)
• Episode 7: ARC-AGI 3 Evaluation Mechanism Under Fire from Developers(2026-07-30, 9 posts)
• Episode 8: Claude Opus ARC-AGI Score Questioned Over API Flaw(2026-07-30, 2 posts)
• Episode 9: ARC-AGI-3 Benchmark Rules Clarified and Official Code Released(2026-07-30, 5 posts)

• [source] How Test Harness Design Impacts ARC-AGI-3 Scores — dkundel · 2026-07-30
• [source] Reddit Debate: Is ARC-AGI 3 an Intentionally Dishonest Measure of AGI? — Glittering-Neck-2505 · 2026-07-30
• Agent Benchmark Harness Criticized: Gimping Long-Term Memory Skews Efficiency Tests — teortaxesTex · 2026-07-30
• Dev Critiques ARC-AGI-3: Already Saturated, Quadratic Scoring Inflates Egos — mgostIH · 2026-07-30
• ML Street Talk: ARC-AGI3 Should Be Benchmarked with a Unified Agentic Harness — burny_tech · 2026-07-30
• Harness Errors Cause 25% Score Swings, Exposing Flaws in Raw Benchmarks — Imaginary_Dinner2710 · 2026-07-31
• [source] ARC-AGI Benchmark Accused of Bad Faith Rigging for Publicity — iruletheworldmo · 2026-07-31
• Agent Performance Shaped by Both Models and Execution Frameworks — LiangSong850509 · 2026-07-31
• Minor Harness Setting Tweaks Radically Alter ARC-AGI Scores — teortaxesTex · 2026-08-01