Why repo-level verification: single-function benchmarks miss real software
dawnsongtweets · x · 2026-08-23
Motivation for Vero: agents are impressive at writing code, but can we trust it at real software scale? Existing benchmarks target single functions or proof generation on a fixed implementation, whereas real verified software (OS kernels, crypto protocols, distributed systems) lives in multi-module repositories where code, specs, and proofs are deeply interdependent—a single refactor can invalidate proofs across the codebase. No prior benchmark evaluates joint code-and-proof generation and formal verification at this scale.
More from Research
- Fortnow: why formalizing complexity results in Lean is hard — proving polynomial time — fortnow · 2026-10-16
- Google's Willow chip cuts quantum errors 40x, raising Q-Day concerns for Bitcoin — jamestagg · 2026-10-16
- A Cookbook for Machine Learning: compiling ML goals into tractable optimization — goyal__pramod · 2026-10-16
- Transluce vets: a year of neuron autointerp work still can't explain complex behaviors — aryaman2020 · 2026-10-16
- Transformer trained on flying bat recordings yields hippocampus 'digital twin' revealing hidden structure — patrickmineault · 2026-10-16
- One year of Recursive Language Models: GPT-5-mini beats GPT-5 by 110% on 132k-token tasks, cheaper too — a1zhang · 2026-10-16