Vero shows the gap: agents rarely build reusable lemma libraries (74% of proof lines)
dawnsongtweets · x · 2026-08-23
Vero results reveal where the capability gap lies: the strongest agent passes 87% of individual specifications but leaves 16 repositories unfinished—remaining specs encode cross-module invariants requiring reusable lemma libraries, which agents rarely build. Across 82 full solves, 74% of proof lines live in helper lemmas shared across specifications. Agents also sometimes abandoned hard-to-verify reference algorithms and wrote simpler implementations satisfying the same specs.
More from Research
- SubQ open-sources Decision LCR, a 72K–115K-token benchmark for decision models — ryanallred · 2026-10-16
- Interpretability debate: 'features' as fundamental neural network units may be wrong — MoonL88537 · 2026-10-16
- FOMO-Dex adds force-aware compliance to dexterous teleoperation for contact-rich robot tasks — HaozhiQ · 2026-10-16
- New mech interp hypothesis: neural networks generalize from the very first training step — burny_tech · 2026-10-16
- Princeton launches research initiative on keeping autonomous AI systems aligned with human intent — LakeBrenden · 2026-10-16
- OrganicHAR at UbiComp 2026: Discovering Activities Sensors Can Actually Distinguish — mynkgoel · 2026-10-16