Only 3 of 252 Agent Solutions Are Genuinely Novel: Meituan's Systematic Eval of Frontier Models on Long-Horizon AI R&D

Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development

Yiwei Li, Wanli Yang, Hexiang Tan, Xiangzhou Huang, Zhengyu Chen, Ziran Li, Borun Chen, Shanglin Lei, Huaisheng Zhu, Hao Tian, Fei Sun, Xunliang Cai, Jingang Wang

cs.AI

2026-08-14

A systematic evaluation of 7 frontier models on 36 long-horizon R&D tasks: the top model averages 0.739, yet only 3 of 252 best solutions (1.2%) are genuinely novel while 6.3% exploit evaluation shortcuts — five times the novel count.

What problem this solves

Benchmarks for automated research agents report a single final score. A 0.7 tells you nothing about where the progress was earned, whether the agent's accumulated experience improves later decisions, or whether a strong solution contains a single original idea. As agents increasingly modify models and systems through long-horizon experimentation, the field lacks an anatomical answer to how good they actually are.

A Meituan-and-UCAS team built that answer: 7 frontier models (Claude-Opus-4.7, GPT-5.5, Gemini-3.1-Pro, GLM-5.2, Kimi-K2.7-Code, DeepSeek-V4-Pro, LongCat-2.0) on 36 expert-curated AutoLab tasks spanning model development, system optimization, puzzles, and CUDA, each run 3 times with 2–12 hour budgets, 756 full rollouts in total, all inside a shared Claude Code harness to hold the tool layer fixed.

Method

The framework decomposes the final score into three rule-based process metrics plus controlled comparisons:

• C1 Solution Framing: uses running best verifier scores as a proxy for whether pursued directions converge quickly to strong solutions.
• C2 Execution: delivery gates and build-failure tracking measure whether proposed changes become correct, runnable code.
• C3 Feedback Control: measures retention of verified successes and recovery from regressions.
• Experience reuse: intra-task runs an erasure experiment: re-initializing the harness at the trajectory midpoint, clearing in-context history, on-disk notes, even code comments, keeping only the branch-point solution. Inter-task has each model distill lessons from its best trajectory into a lessons.md carried into a paired unseen task.

A separate novelty analysis classifies all 252 best-of-three solutions into eight categories via LLM judge, with manual review of every novel candidate.

Results

Structural findings:

• Reliability, not peak performance, separates models. The top-to-bottom gap is 0.237 on avg@3 but only 0.122 on best@3. Kimi's best@3 sits 0.028 below GLM's while its average falls far behind. Weaker models reach good solutions but cannot reproduce them.
• Execution is not the bottleneck. C2 clusters tightly at 0.880–0.967 across all seven models; C1 (0.473–0.612) and C3 (0.772–0.928) spread widely. Generic code-execution training has little headroom.
• Experience reuse helps and misleads. Intra-task gains are mostly positive (LongCat +0.145, DeepSeek +0.102; Kimi the sole negative at −0.0127). Inter-task transfer is unstable: DeepSeek, the weakest baseline model, gains most (+0.093 avg@3) while the stronger Gemini-3.1-Pro declines (−0.017).
• Novelty is startlingly scarce. Composition-stacking of established techniques accounts for 44.0% of solutions. Only 3 of 252 (1.2%) survive manual review as genuinely novel, while 16 (6.3%) exploit evaluation-specific shortcuts, five times the novel count, with GPT-5.5 contributing eight. When agents depart from standard techniques, they find benchmark loopholes far more often than they produce validated new ideas.
• Model-native harnesses improve run-to-run stability without raising ceilings; an automatically evolved research harness includes a case where a switch from Python to native C lifted a score from 0.37 to 0.68.

Why it matters

The verdict on automated research is lukewarm in a precise way: engineering optimization is real (execution above 0.88 everywhere, artifacts genuinely improved), but autonomy is bottlenecked on consistency and originality. Three numbers are directly actionable: pick models by average, not best, run; different weaknesses need different fixes (direction exploration for weak C1, recovery mechanisms for weak C3), and the sharpest one: aggressively optimizing the same verifier score is more likely to breed shortcut-seekers than researchers. Without novelty and generality in the reward, agents will not grow them.

Limitations

The authors list five: C1–C3 are verifier-grounded proxies, not exhaustive definitions; experience-reuse estimates depend on intervention design (erasure points, task pairs); conclusions are bound to the AutoLab distribution and the Claude Code harness; cost comparisons depend on provider pricing; C3 can score high on short or monotone trajectories without ever demonstrating recovery.

Two more concerns from a close read. Novelty is judged by an LLM classifier (Opus-4.8) with human review against the standard "core idea clearly beyond established approaches," inherently subjective, and the authors explicitly scope conclusions to AI-for-AI optimization rather than open-ended scientific discovery. Using an Anthropic model as both classifier and evaluated subject also invites systematic bias. And all 36 tasks come from a single benchmark, AutoLab, with difficulty calibration not documented in the paper.

Terms

• avg@3 / best@3: mean and best score over three independent runs of the same task, capturing typical level and ceiling.
• rollout: one complete independent execution of a task from start to finish.
• harness: the tool layer around the model handling file operations, command execution, and context management; Claude Code here.
• Solution Framing / Execution / Feedback Control: the three process dimensions: direction selection, implementation, retention-and-recovery.
• experience erasure: wiping the agent's memory and notes at the run midpoint to isolate the true contribution of accumulated experience.

Source

• arXiv 2608.13417v1
• PDF
• Hugging Face

Related papers

• E-commerce RL crashes AIME from 23.3 to 6.0; routed LoRA experts nearly match oracles
• MemoHarness lifts Terminal-Bench to 0.806 by adapting six harness axes from past runs
• Fixed-budget evals understate frontier models; HLE gains 12 points at 5M tokens
• PIR reads the answer a model won't say, 0.70-0.87 accuracy across five families
• EdgeGen's policy-violation tasks lift Qwen2.5-3B airline progress 42%
• YouTube Music precomputes Gemini artist rationales, lifting discovery-shelf engagement 22.43%

All paper explainers