TLAPS-Bench: 265 real-world TLA+ proof targets; top model GPT-6.1 Sol solves only 44.3%
tianyin_xu · x · 2026-11-04
TLAPS-Bench (arXiv 2611.02849) is a new benchmark and execution harness for agentic AI formal verification via the TLA+ Proof System.
- 265 proof targets from 95 real-world system TLA+ specifications, covering cache coherence, Byzantine consensus, and write-ahead logs — properties never proved before
- The harness supports iterative proof development with proof tools/libraries and safeguards against reward hacking
- Evaluated frontier agents on 70 targets across 12 specs: best model GPT-6.1 Sol (max) proved 31 (44.3%) at pass@3 and surfaced 46 bugs
Paper and code are open.
More from coding & agent
- Cua Driver integrates with Google Antigravity for background cross-app computer agents — m4rkmc · 2026-11-04
- Sentdex: OpenHands was the first reasonably useful agent to abstract away the code editor — rajistics · 2026-11-04
- Security advisory: run AI agents in managed cloud environments, not employee endpoints — amasad · 2026-11-04
- Apple's App Intents lets Siri AI control third-party apps with Agent Skills support — op7418 · 2026-11-04
- Coding agents more than double every released speculative decoding system, DFlash2 hits 10.87x — berkeley_ai · 2026-11-04
- Game dev pushes back on 'full game in 24 hours with AI' claim: it's a demo, not a game — AIandDesign · 2026-11-04