Reasonable post-trains Nemotron 3.5 on 3,000 Verus proofs, lifting pass@1 ~30 points to 73.3%
fhuszar · x · 2026-10-16
Ferenc Huszár's team at Reasonable, with early access to Nemotron 3.5 Super VL from NVIDIA, released tlaverus-train, a dataset of 3,000+ challenging agentic Verus proof trajectories derived from TLA+ temporal-logic specifications, plus the harder tlaverus-60 benchmark.
- Goal: teach models to interact with verification tools, diagnose failures, and iteratively repair proofs — agentic formal verification, not isolated proof generation.
- Results: post-training lifted pass@1 by 30 percentage points to 73.3%, with gains transferring to unrelated Verus benchmarks while preserving general knowledge and coding ability.
- Takeaway: specialized open-weight models can make formal verification practical for everyday software engineers.
Related event: Fine-Tuning Nemotron on 3,000 Verus Proofs Boosts Pass Rate to 73%(2 posts)→
More from Models
- Nathan Lambert slams Meta's Muse Open License as 'trash' — natolambert · 2026-10-16
- Google launches Gemini 4 Argon to Ultra and Pro subscribers, touting physics and 3D skills — ShikharMurty · 2026-10-16
- Google gives X users early access to test Argon, not just companies — lyraxana · 2026-10-16
- Grok Voice 3.0 Tops Design Arena Speech Leaderboard, Up 7.4 Points — XFreeze · 2026-10-16
- Robotics LLMs tested on 16 tasks: GPT-6 Astra hits 17.2% success vs Fable's 9.1% — pabbeel · 2026-10-16
- Meta open-sources Muse Spark: top US open model, scores just below DeepSeek — bindureddy · 2026-10-16