FrontierCS 2 Launches: Agents Must Recreate and Beat Paper Methods from Stripped Code

Qiuyang Mang's team released FrontierCS 2, a benchmark evaluating LLM agents' ability to do open-ended computer science research and development. Even the strongest LLM agents pass only about half the tasks, showing frontier CS R&D capability is far from saturated.

Confirmed

• The benchmark contains 48 tasks across 13 CS fields, all built from published papers, with domain experts refining each task design
• Task construction: take the paper authors' official open-source code, delete the core contribution, and the deleted code serves as the human baseline
• Agents are placed in a realistic research setting: inheriting a codebase, they must implement the method from scratch without reading the paper or accessing the internet, and surpass the original method on evaluation metrics chosen by the paper's authors
• The team likens this design to a harder, research-grade evaluation (@lateintersection described it as a step beyond Kaggle-style contests); @ChengleiSi noted the benchmark is positioned against Kaggle and related evaluations from Google DeepMind
• The benchmark is powered by Modal

Why it matters

• Traditional benchmarks mostly feature closed-form QA or programming tasks with clear answers, while real research requires working with incomplete information and open-ended goals; FrontierCS 2 directly measures research productivity by requiring agents to "reproduce and surpass published contributions"
• The 50% pass rate of the strongest models offers a clear improvement target and a yardstick for progress on AI research agents

2026-10-23 ~ 2026-10-23 · 5 related posts

• [source] FrontierCS 2 benchmark asks agents to reimagine paper code from scratch and beat the original — OfirPress · 2026-10-23
• FrontierCS 2 launches with 48 expert-refined tasks to benchmark AI research agents — ChengleiSi · 2026-10-23
• Laude Institute launches FrontierCS 2: agents must beat deleted paper contributions blind — lateinteraction · 2026-10-23
• [source] FrontierCS 2: top LLM agents solve only ~half of open-ended CS R&D tasks — Qiuyang Mang · 2026-10-23

1 near-duplicate retellings: blelbach