Vero shows the gap: agents rarely build reusable lemma libraries (74% of proof lines)
dawnsongtweets · x · 2026-08-23
Vero results reveal where the capability gap lies: the strongest agent passes 87% of individual specifications but leaves 16 repositories unfinished—remaining specs encode cross-module invariants requiring reusable lemma libraries, which agents rarely build. Across 82 full solves, 74% of proof lines live in helper lemmas shared across specifications. Agents also sometimes abandoned hard-to-verify reference algorithms and wrote simpler implementations satisfying the same specs.
Related event: Vero Benchmark: GPT-5.5 Fully Verifies Only 27/43 Repositories(2 posts)→
More from Research
- Research: Misconfigured Admin Prompts Can Invert LLM Safety Layers — Simple_Passion_7741 · 2026-08-23
- 7,500-line interactive textbook teaches building LLMs from scratch — tom_doerr · 2026-08-23
- ProteinDPO Adopts DPO to Align Protein Models with Experimental Stability — bravo_abad · 2026-08-23
- Article: Ontology evolution from semantics to AI agents — adnan_hashmi · 2026-08-23
- DelveRL: Open-Source Roguelike Built Specifically for Training Game-Playing Agents — SnyderConsulting · 2026-08-23
- Vero includes formal audit mechanism for machine-checked spec errors — dawnsongtweets · 2026-08-23