Verus Proof Index: new formal verification benchmark ranks 25 models, GPT-6 Astra leads at 59.1
fhuszar · x · 2026-11-17
The Reasonable team released the Verus Proof Index, the most comprehensive formal verification benchmark for software to date: 25 models act as autonomous agents generating proofs that pass the Verus verifier across 100 problems.
- Problems: verified systems code, distributed protocols transpiled from TLA+, and smaller programs, drawn from public VerusBench/VeruSAGE-Bench plus the private tlaverus dataset (training contamination excluded for the latter)
- Scoring: beyond correctness, the index measures efficiency—tokens and turns needed per passing proof, since "eventually correct" is only part of usefulness; one run per problem, so 1-2 point gaps can be noise; every proof independently checked for integrity violations
- Leaderboard: GPT-6 Astra (OpenCode) tops at 59.1, solving 94/100 for $224; Claude Fable 5.1 (58.7) and Claude Opus 5.5 (58.2) follow, with Opus 5.5 posting the best correctness and no integrity violations at just $232; Gemini 4 Argon has the largest efficiency bonus (+4.6) but costs $543
First in a series of benchmarks quantifying agentic proof-generation capability.
Related event: First Verus Formal Verification Benchmark Released, GPT-6 Astra Tops(2 posts)→
More from coding & agent
- AI engineering job-hunt workshop: scrape real JDs, build portfolio, tailor CV with an AI pipeline — Al_Grigor · 2026-11-17
- Rust MCP code-search server benchmarked against ripgrep on Grafana's 24k files — Designer-Seaweed4661 · 2026-11-17
- T3 Code hands-on: a multi-harness coding client that swaps between Codex and Claude Code — vista8 · 2026-11-17
- Hermes agent joins Zoom/Meet/Teams calls, records and writes follow-ups — alexcovo_eth · 2026-11-17
- Blogger finds Astra skips steps on research tasks while carbon dutifully finishes them — lxfater · 2026-11-17
- AWS open-sources local emulator letting coding agents test 200+ services without an account — bestvicam · 2026-11-17