New Vero Benchmark Shows Top Agent Fully Verifies Only 63% of Repos
shizhediao · x · 2026-08-24
Dawn Song's team introduced Vero, the first benchmark for evaluating AI agents on joint implementation and proof synthesis at the repository level. As AI agents write a growing share of software, working code is no longer sufficient; machine-checked guarantees of correctness and security are required.
Key Findings:
- Repository-scale verified code generation remains out of reach: the strongest frontier agent fully verifies only 27 of 43 real-world repositories (63%).
- Defines a new task of joint implementation and proof synthesis.
More from coding & agent
- Are the fastest GPU kernels now written by AI? Kernels are hyper-verifiable, argues one dev — fleetwood___ · 2026-10-16
- Anthropic lead engineer's 4-question prompt makes agents more ambitious, now a free skill — daniel_mac8 · 2026-10-16
- ChainForge dev shows 'feature map' for orchestrating subagents on large PRs — IanArawjo · 2026-10-16
- aiDotEngineer benchmarks agent orchestration pipelines, staging humans as Foreman and workers — vikvang1 · 2026-10-16
- Developer mass-produces 200 videos in hours using Claude Code + Hyperframes — mcraddock · 2026-10-16
- Penny University hosts software factory discussion; dev shares $40/mo vendor-agnostic AI stack — JnBrymn · 2026-10-16