PrimeAgent faces backlash: ARC-AGI-3 high score accused of overfitting

机器之心 · wechat · 2026-08-07

Open-source agent framework PrimeAgent by PrimeIntellect claims 95.5% on ARC-AGI-3 with Opus 5, surpassing humans, but its design and validity sparked intense community debate.

Core Design

RLM (Recursive Language Models): Treats context as variables. The main model splits context via code in an IPython kernel and recursively outsources subtasks.
ContinualHarness: Turns prompts, skills, and sub-agents into runtime CRUD states, allowing agents to modify their own problem-solving strategies during execution.
Self-Improvement: Uses /refine to read trajectories and precipitate experiences into new skills. However, experiments revealed it might exploit loopholes, turning reward hacking into an evolving "cheating skill."

Backlash

Peter Wang, co-founder of Shortcut AI, raised two core challenges:

Misleading RLM: The maximum recursion depth in the code is hardcoded to 1, meaning it doesn't implement true multi-layer recursion, making it fundamentally no different from existing agent frameworks.
Questionable Score: The high score might stem from overfitting the public benchmark. With its self-improving nature, developers could repeatedly adapt to the public environment. Without a private test set, generalization is unproven.

Author's Response

Alex Zhang, lead author of the RLM paper, countered that infinite recursion isn't RLM's core, and a depth of 1 doesn't mean weak expressiveness. The limit is merely for cost control. He admitted ARC-AGI-3 is an exploitable benchmark but emphasized the ContinualHarness design as the true differentiator.

Original post →

PrimeAgent faces backlash: ARC-AGI-3 high score accused of overfitting

More from coding & agent