Vero: First Benchmark for Repository-Scale Formal Verification by AI Agents
dawnsongtweets · x · 2026-08-23
Dawn Song's team introduces Vero, the first benchmark for joint implementation and proof synthesis at the repository level to evaluate AI's ability to build formally verified software. It includes 43 multi-module Lean 4 instances with 743 APIs and 2,705 specifications. Evaluations reveal that even the strongest frontier agent (GPT-5.5 at high reasoning effort) fully verified only 27 of 43 repositories within 90 minutes. The study identifies the construction of reusable lemma libraries for cross-module invariants as a major capability gap.
More from coding & agent
- Vercel Goes Full Backend: Python GA, Containers, and Daemons — cramforce · 2026-10-16
- WorldMemArena Accepted at NeurIPS: Memory Storage ≠ Memory Use — xwang_lk · 2026-10-16
- Vercel Ship Demos Container Deploys for Go, Ruby, Java and More — edwin · 2026-10-16
- TRL adds day-0 support for fine-tuning Muse Spark Open's 816B MoE with LoRA on 64 H100s — QGallouedec · 2026-10-16
- Databricks engineers land impactful PRs on day 2 with AI dev tools, vs week 3 before — Yuchenj_UW · 2026-10-16
- Perplexity open-sources SPACE, the sandbox platform powering its Computer agents — AravSrinivas · 2026-10-16