FULL STORY

ARC-AGI-3 Scores Spark Controversy and Reflection

High scores on the ARC-AGI-3 benchmark sparked cheating and overfitting controversies. Amid developer skepticism, officials clarified rules and open-sourced the codebase, prompting industry reflection.

2026-07-25 ~ 2026-08-01 · 9 episodes · 64 posts

Episode 1 · Opus 5's High ARC-AGI-3 Score Sparks Cheating and Overfitting Controversy (2026-07-25, 11 posts)

Claude Opus 5 scored 30.2% on the ARC-AGI-3 benchmark, outperforming peers by 3 times. However, this high score immediately triggered widespread allegations of cheating and overfitting within the AI community, with multiple users and industry insiders questioning its true capabilities and sparking reflections on the validity of evaluation systems and their ties to commercial interests.

Confirmed

It is confirmed that Opus 5's specific score is 30.2%, and this result has caused a severe crisis of confidence in the industry. Reddit user @sdnr8 questioned the opacity of closed-source models, suggesting the high score might not come from "pure model" capabilities but from external proxy frameworks like loop or harness. User @VraserX directly accused Anthropic of "cheating," claiming they specifically trained for the benchmark's puzzle patterns, converting visual reasoning into explicit algebraic problems. Furthermore, a person claiming to be from a frontier AI lab told @flowersslop that the score "looks fake," implying either excessive benchmark gaming or a severe underestimation of Anthropic's lead by competitors. Chris Szegedy (@ChrSzegedy) and @DrSingularity explicitly stated that ARC-AGI has little to do with AGI, though the latter acknowledged that AI is indeed getting stronger. Gary Marcus also emphasized that scoring high on benchmarks does not equate to approaching AGI, arguing that the score improvement is more likely due to targeted optimization rather than the true generalization of abstract reasoning capabilities.

Unconfirmed

Whether Opus 5 actually used external proxy frameworks, or whether Anthropic indeed conducted targeted specialized training, currently remains at the stage of community speculation and accusations, with no substantial technical evidence provided for the relevant claims.

Why it matters

This controversy affects more than just Anthropic's reputation; it reflects the industry's deep-seated concerns about the validity of AI evaluation systems. On one hand, as relayed by @morqon, rapidly saturating evaluations lose their discriminative power, with some believing the benchmark was nearly "solved" on day one. On the other hand, discussions forwarded by @burnytech indicate a growing default assumption that ARC-AGI progress mainly comes from targeted RL environments, prompting calls for ARC-AGI-4 to reduce public demos to prevent model overfitting. @JasonBotteril also pointed out that naming a benchmark directly after AGI is almost destined to induce labs to optimize for it specifically. In addition, @rbhar90 emphasized that the "strong reasoning ability" narrative of frontier labs is deeply bound to huge commercial interests, advocating for more open, verifiable benchmarks (such as the self-built Witness test set), and noting that Opus 5's improvement on ARC-AGI-3 did not transfer to other test sets. @inductionheads also noted that if a model is indeed trained in an RL environment similar to ARC-AGI, its performance cannot prove "generalization." This indicates that existing public benchmarks are facing severe methodological challenges.

Episode 2 · Deep Dive into Opus 5 Hidden Reasoning and ARC-AGI Score (2026-07-25, 2 posts)

Technical analyses explore Opus 5's hidden reasoning mechanisms and agent-like behaviors, while subsequent clarifications detail its 30.16 ARC-AGI-3 score using the RHAS evaluation and compute trends.

Episode 3 · Anthropic's Benchmark Scores Spark Community Trust Crisis (2026-07-25, 2 posts)

Discussions around Anthropic's benchmark scores have sparked a trust crisis within the community. Despite decent results that reportedly outperform Opus 4.8, users tend to question the benchmark's reliability, with high ECI metrics further amplifying score discrepancies and confusion.

Episode 4 · Gary Marcus Says ARC-AGI Name Is Misleading (2026-07-27, 2 posts)

Gary Marcus and others argue that the name "ARC-AGI" is misleading, as it makes people mistakenly believe the benchmark directly tests for AGI, a concept that remains vaguely defined and highly debated.

Episode 5 · Human Baselines Missing in AI Evaluations, Highlighting Human-AI Synergy (2026-07-29, 4 posts)

Experts warn that complex AI benchmarks are losing crucial human baselines for comparison. To address this, researchers are advocating for new evaluation standards like Stanford's CollabSkill framework, which focuses on measuring the collaborative performance of humans and AI agents in real-world tasks.

Episode 6 · Optimized Memory Settings Triple GPT-5.6's Score on ARC-AGI-3 (2026-07-30, 27 posts)

OpenAI officially confirmed that by enabling "reasoning retention" and "context compaction" in the Responses API, GPT-5.6 Sol's score on the ARC-AGI-3 public test set surged from 13.3% to 38.3% (a 188% increase), reaching SOTA levels while reducing token consumption to 1/6th of the original. This reveals that the memory management mechanism within the test harness, rather than pure model reasoning ability, is the key determinant of performance in complex long-horizon tasks.

Confirmed

  • GPT-5.6 Sol initially performed poorly on the ARC-AGI-3 benchmark, an issue first highlighted in tests by developer @sandersted.
  • By enabling two specific API settings used internally by ChatGPT and Codex, the model's score increased roughly threefold (from 13.3% to 38.3%), alongside a sixfold improvement in token efficiency (reducing consumption to 1/6th).
  • The two key settings are reasoning retention and context compaction, which allow the model to remember previous thought processes and operate continuously across multiple context windows.
  • OpenAI engineer @ilanbigio noted that standard test harnesses discard the model's reasoning process after each step and lose early operation records when the context fills up, limiting the model's evaluated performance in isolated environments.

Why it matters

  • This discovery challenges the current industry-standard isolated evaluation methods. Developers like @Angaisb pointed out that complex tasks can be easily solved by optimizing the test harness, meaning that merely comparing raw benchmark scores does not truly represent the actual intelligence of different AI systems.
  • The deep synergy between product frameworks (like ChatGPT and Codex's internal mechanisms) and model capabilities has a decisive impact on complex task performance, indicating that the performance bottleneck often lies in context management rather than the model itself.

7 more related posts →

Episode 7 · ARC-AGI 3 Evaluation Mechanism Under Fire from Developers (2026-07-30, 9 posts)

Recent developers have strongly questioned the fairness and evaluation mechanism of the ARC-AGI 3 benchmark, arguing that the current test framework design, scoring rules, and even testing motives are severely distorted and cannot truly measure general AI capabilities.

Confirmed

  • Framework limits memory: Developers @Glittering-Neck-2505 and @teortaxesTex point out that the current unified test framework deliberately prevents reasoning agents from maintaining context across operations, forcing models to constantly "forget." This practice, aimed at cross-provider standardization (using the same completion-style endpoint and not passing back reasoning logs), limits agent long-term memory and fails to effectively measure true model efficiency.
  • Evaluation framework significantly affects scores: Developers @ImaginaryDinner2710 and @ilanbigio (via @dkundel) note that model scores heavily depend on the test harness. For ARC-AGI, an OpenAI model scored only 13% due to improper invocation, jumping to 38% after correction. Additionally, just two adjustments to the harness settings led to a significant improvement for Claude 3.5 Sonnet. Properly tracking reasoning and using context compression are crucial for achieving true high scores.
  • Scoring mechanism criticized: Developer @mgostIH found that current harnesses have saturated ARC-AGI-3; a small amount of reinforcement learning (RL) can greatly boost scores. He also strongly criticizes the benchmark's use of squaring the percentage of solved problems, calling it baseless and "myth-making."
  • Alleged malicious tuning: Developer @iruletheworldmo harshly criticizes ARC-AGI as a catalyst for public attention, suggesting the testers may have maliciously lowered model scores to create the illusion that the benchmark is extremely hard to saturate.
  • Unfair base model testing: ML Street Talk (via @burnytech) argues that modern AI is essentially a hybrid neuro-symbolic system, and a single-context reasoning window is insufficient for adaptation, so directly testing base models on ARC-AGI3 is unfair.

Unconfirmed

  • Whether testers subjectively "maliciously" tuned parameters to create the illusion of extreme difficulty (currently only a harsh accusation by @iruletheworldmo).
  • Whether some models used a different harness in internal preview tests than in public evaluations (question raised by @LiangSong850509).
  • Whether the industry can reach consensus on establishing and adopting a unified agent framework for testing different models (currently only a suggestion from ML Street Talk).

Why it matters

  • Affects benchmark credibility: If the test mechanism can be easily gamed with a small amount of RL and saturates, the scoring formula is controversial, and there is even potential for artificially lowering scores, the benchmark's authority will be greatly diminished.
  • Evaluation must reflect real usage: Developers emphasize that in real development scenarios, models always run with specific agent frameworks. When designing cross-platform standardized tests, one should not sacrifice core agent capabilities (like long-term memory). The industry urgently needs to abandon "bare" benchmarks.

Episode 8 · Claude Opus ARC-AGI Score Questioned Over API Flaw (2026-07-30, 2 posts)

A developer argued that Claude Opus's high ARC-AGI score was inflated by an API flaw, but community members maintain that its generalization capabilities still outperform GPT-5 under the same testing framework.

Episode 9 · ARC-AGI-3 Benchmark Rules Clarified and Official Code Released (2026-07-30, 5 posts)

Recent evaluation methods for large models on the ARC-AGI-3 benchmark have sparked controversy. In response, ARC-AGI founder François Chollet and official member Greg Kamradt have clarified the evaluation rules and API usage, and open-sourced the official testing codebase to ensure fairness and transparency across model evaluations.

Confirmed

  • Custom tools banned: François Chollet explicitly prohibits using custom tools developed specifically for this benchmark or systems containing knowledge of the test format and content.
  • Fair testing mechanism: Addressing the controversy over Claude Opus, Greg Kamradt clarified that the official testing used the exact same "sliding window" convention for both Anthropic and OpenAI models.
  • Open-source evaluation tool: Greg Kamradt shared the official ARC Prize open-source codebase arc-agi-3-benchmarking, which provides complete environment setup and running guides, supporting configuration of various API keys to evaluate large models on complex abstract reasoning tasks.
  • API field clarification: Greg Kamradt corrected misconceptions about the optional reasoning field in the API, emphasizing it is not for extracting the model's internal chain-of-thought (CoT) but only for recording and associating external outputs.

Why it matters

A unified and open evaluation standard helps eliminate suspicions of "cheating" among model providers during benchmarking. Explicitly banning custom tools and publishing test code provides developers with a reproducible evaluation environment, further solidifying ARC-AGI's authority as a benchmark for frontier model intelligence.