ETH's AuXSpec: executable postconditions as RL reward, 27B model beats frontier LLMs at SpecGen
eth-algopt · hf · 2026-11-18
ETH researchers introduce AuXSpec, a Lean framework tackling the formal-specification bottleneck:
- Method: pairs declarative postconditions with executable counterparts (via type class resolution when possible, otherwise defined separately with equivalence proofs); test cases evaluate declarative specs by execution, and agreement serves as an RL reward — avoiding expensive proof-based feedback
- Data: 3,037 synthesized LeetCode problems, expanded to 19.5K training samples via expert iteration on Codeforces
- Results: training Qwen3.5-9B and Qwen3.8-27B lifts SpecGen pass@4 on AuXSpec-Test from 0.7%→82.7% and 14.3%→84.3%; on out-of-domain Verina, 7.0%→90.5% and 21.9%→90.5%, with the 27B model outperforming all frontier models; pass@1 beats prior open-source SOTA VeriSpecGen-30B-A3B (73.7% and 84.5% vs 47.6%)
- Released: model weights, 3,037 problems with test cases, and a 300-problem AuXSpec-Test benchmark with expert review and correctness proofs
More from coding & agent
- Safer coding agents: synthetic credentials via 1Password intercepting proxy — craigbalding · 2026-11-18
- Which web search API do you actually use for agents? Exa, Tavily, Brave compared — Fluffy-Resident-8734 · 2026-11-18
- AI short film workflow revealed: Seedance 2.5, Kling 4.0, Nano Banana, Suno, Premiere — Uncanny_Harry · 2026-11-18
- Microsoft adds role-based Copilot skills as Agent 365 passes 50 million agents — shashib · 2026-11-18
- philschmid: replace built-in multi-agent features with an MCP server of the harness itself — _philschmid · 2026-11-18
- Running multiple LLM agents 24/7 on a VPS: why Telegram management falls apart — thetapereader · 2026-11-18