Lean proof benchmark accused of leaving kernel-bug and run_meta loopholes open
teortaxesTex · x · 2026-09-25
A discussion of an AI math proof study: because the task lacked certain Mathlib theorems (e.g. Ising/Szegő), it was read as expecting exploit discovery — blocking naive sorry/axiom shortcuts with forbidden lists while leaving Lean kernel bugs, runmeta and root-level Lean loopholes open.
The poster questions the reasoning, highlighting how ambiguous such adversarial robustness test designs can be.
More from Research
- Study: Reasoning hurts 15.7% of multimodal embeddings; training-free SURE router fixes it — _reachsumit · 2026-09-25
- Google scales learned cross-task relationships in YouTube's production recommender — _reachsumit · 2026-09-25
- ByteDance's OneTrans-V2 unifies retrieval, pre-rank and fine-rank, lifting GMV 9.74% — _reachsumit · 2026-09-25
- Training-free Seek framework beats BM25 by 82% on BRIGHT via self-evaluative iteration — _reachsumit · 2026-09-25
- ByteDance's X-Rec does generative retrieval via flow matching with 3.46x throughput — _reachsumit · 2026-09-25
- Multi-agent AI teams are swayed by deceptive agents even in the minority, paper finds — sethlazar · 2026-09-25