Stanford paper: roundtrip verification catches unfaithful LLM autoformalization with 1.4-2.5x NLI drift
StanfordAILab · x · 2026-10-20
A Stanford-linked arXiv paper tackles a key open question in autoformalization: when an LLM translates natural language into formal, machine-checkable representations, how do we know the output is faithful?
Roundtrip verification and repair
- Formalize a statement, translate the result back to natural language, re-formalize it, and use a formal tool to check logical equivalence—no ground-truth annotations required.
- Agreement between the two formalizations is evidence of faithfulness; disagreement triggers stage-level diagnosis that localizes the error to a specific translation step, followed by a scoped repair operator.
Results
- Evaluated on two statutory domains (Texas Transportation Code, Texas Parks and Wildlife Code) with Claude Opus 4.6 and GPT-5.2 against three repair baselines.
- Diagnosis-guided scoped repair was most effective, contingent on diagnosis reliability; across both domains and models, rules failing the equivalence check showed 1.4x-2.5x more NLI drift than rules passing it.
More from Research
- DreamZero debuts Dream-1: zero-shot robot brain beats Opus 5.5 at 16x lower cost — gharik · 2026-10-20
- OmicsBench v1: 784 omics tasks benchmark 31 model-harness combos, frontier models tie — kenbwork · 2026-10-20
- DiscoveryRSI: 27B model hits 6.3% on post-cutoff scientific hypotheses, beating GPT-6 — omarsar0 · 2026-10-20
- CERN and Mistral partner to build the first foundation model for collider physics — Upset-Jellyfish-235 · 2026-10-20
- Argon autonomously processes Hubble and Chandra archival FITS data into astronomy visuals — fofrAI · 2026-10-20
- Open-source JEV-gated adaptive thinking makes LLMs reason only when unsure — dimknaf · 2026-10-20