FrontierCS 2: benchmark makes AI agents beat published CS research blind
arena · x · 2026-11-18
UC Berkeley PhD candidates MangQiuyang and lihanc02 joined Arena to introduce FrontierCS 2, a benchmark challenging AI agents to outperform published CS methods without seeing the paper or web access. They argue reliable benchmarks require more than good tasks: defending against reward hacking and robust eval infrastructure matter equally for open-ended research tasks. The discussion covers evaluation exploits like bypassing RL with search and reusing cached graph-algorithm results, and why human research judgment still matters.
More from coding & agent
- SilentGuard: harness-level invariant checks cut agent API silent-failure errors from 74% to 1.7% — Genentech · 2026-11-18
- Meta AI's Self-Improving Memory: agents learn memory skills into their weights — faeze_brh · 2026-11-18
- One PowerShell cmdlet, New-AgentHarness, spins up an OpenAI-powered agent — dfinke · 2026-11-18
- Doodlestein Shares His Multi-Agent Coding Stack: agent-mail + beads + ntm — doodlestein · 2026-11-18
- The bitter lesson is coming for software engineering: let agents discover abstractions — tylerjdunn · 2026-11-18
- MCP Agent Mail: open-source shared inboxes and coordination for multiple AI coding agents — doodlestein · 2026-11-18