TLAPS-Bench author clarifies: passing formal verification doesn't mean bug-free implementations
moarbugs · x · 2026-11-05
In a thread about TLAPS-Bench, the author clarifies that the benchmark only validates properties at the specification level: TLA/TLA+ is not about system implementations, so even if agents can prove safety/liveness properties via TLAPS-Bench, that does not guarantee the systems are bug-free. He admits "abusing" TLA+ to describe implementations in Specula, noting the authentic flow is to get the spec right first, then produce code — which is itself nontrivial. The questioner suggests clarifying this in the paper/README, citing widespread "snake oil" around formal methods.
More from Research
- CoT controllability far higher than evals suggest: self-instructions beat prompts, can evade monitors — kastnerkyle · 2026-11-05
- Formal methods assurance levels debated: TLAPS-proven properties can still fail in real systems — moarbugs · 2026-11-05
- Perceptron's egocentric video labeling system boosts hand reconstruction accuracy 12-25% on HOT3D and ARCTIC — SucceededMind · 2026-11-05
- TxBench–Antibody Development: 60 verifiable evals across 7 drug-development stages — kenbwork · 2026-11-05
- Stanford open-sources frax: JAX robot kinematics with IK over 100k FPS, no GPU needed — ZeYanjie · 2026-11-05
- Yale presents 4 AI cardiometabolic disease studies at AHA26, 3 with simultaneous publications — BrdKhsrv · 2026-11-05