8 of OpenAI's Lean formalization challenges are broken and trivially hackable
gklambauer · x · 2026-10-09
A blog post by Timeroot reveals that at least 8 Lean formalization challenges in OpenAI's repo are hackable.
The core flaw: definitions that should be fixed are listed in the definitionnames field of the . config, so they can be freely overridden without affecting Comparator's acceptance. In EuclideanFiveColor.lean, redefining ProperColoring as False makes the theorem trivially provable and still accepted.
Affected challenges include Bernier, LogspaceEquality (L=RL=BPL), KServer, Naimark, OccupiedOverlap, Rokhlin, and SpinAngle. The author believes this hasn't been publicly discussed before and warns some cases are harder to spot than others.
More from Research
- Stepped MoE paper: one model scales from 1B to 4B parameters, beating dense counterparts by 2-5% — pmttyji · 2026-10-09
- Palisade study shows o1-preview and DeepSeek R1 hack chess games rather than lose — burny_tech · 2026-10-09
- DLoop: looped speculative decoding cuts target-model passes, boosting speedup 5-41% across EAGLE-3 and more — pmttyji · 2026-10-09
- SatNav: Scalable Long-Horizon UAV Vision-Language Navigation Benchmark From Satellite Imagery — Jiajun Jiang · 2026-10-09
- Tencent Hunyuan Maps the Geometry of RLVR in LLMs, Releases Alpha-Stabler Framework — Tencent-Hunyuan · 2026-10-09
- Harrison Chase: trajectory labeling is several questions, not one pass/fail — Jev lands in LangSmith evals — hwchase17 · 2026-10-09