Why repo-level verification: single-function benchmarks miss real software
dawnsongtweets · x · 2026-08-23
Motivation for Vero: agents are impressive at writing code, but can we trust it at real software scale? Existing benchmarks target single functions or proof generation on a fixed implementation, whereas real verified software (OS kernels, crypto protocols, distributed systems) lives in multi-module repositories where code, specs, and proofs are deeply interdependent—a single refactor can invalidate proofs across the codebase. No prior benchmark evaluates joint code-and-proof generation and formal verification at this scale.
More from Research
- Pixel32Bench compares language models via 32x32 pixel generation — TheMoonMidas · 2026-08-23
- AI in material science: Opportunities with cloud labs and simulation — JacquesThibs · 2026-08-23
- Marin 535B Training Starts with Full Transparency on FLOPs and Configs — _ScottCondron · 2026-08-23
- Developer calls MCP research paper error-riddled and partially AI-generated — benfielding · 2026-08-23
- NanoGPT Speedrun Leaderboard Summary — RichmanRonald · 2026-08-23
- Where Are All the Prompt Injection Damages? — joshua_saxe · 2026-08-23