FrontierCS 2: top LLM agents solve only ~half of open-ended CS R&D tasks
Qiuyang Mang · hf · 2026-10-23
FrontierCS 2 is a new benchmark of 48 open-ended CS R&D tasks for LLM agents: it takes a paper and its released code, deletes the paper's contribution, and uses that deleted code as the human reference bar. Unlike prior open-ended benchmarks, the agent works inside a live research repo and may submit patches at any point, with a judge returning development-suite scores while the evaluator and held-out final suite stay private. Even the best agents pass only about half of the tasks.
More from coding & agent
- Garry Tan endorses Aside Browser as the fix for agents getting blocked on the web — garrytan · 2026-10-23
- 1Password launches tool letting AI agents log in for you without seeing your password — AndrewSchmidtFC · 2026-10-23
- MCP server turned rogue 30 days after approval, prompting open-source mcp-guard lockfile tool — ErgoniaWorks · 2026-10-23
- Test: Coding Agents Won't Use an MCP Tool Unless AGENTS.md Mentions It — colinmcnamara · 2026-10-23
- Open-Source Tennis Models Plus Claude Turn Video Into a World-Class Coach — tweetsatpreet · 2026-10-23
- awesome-ai-apps: 132 Free Open-Source AI Agent and RAG Projects on GitHub — Arindam_1729 · 2026-10-23