OpenFrontiers benchmark: LLM web-enabled "discoveries" mostly fetch published solutions
geoffwolfe · x · 2026-11-02
UIUC researchers and collaborators released OpenFrontiers, a resource for measuring whether LLMs genuinely rediscover scientific results or just retrieve published answers. The key point: a valid proof doesn't tell you whether an agent derived it or found it online.
What's in it
- Corpus: 12,151 cross-disciplinary open research problems with statements, context, and progress notes.
- Benchmark: 200 formal math problems (110 resolved in 2026, 90 still open), each with a Lean 4 statement checked by automated probes and two independent reviews; a quality-filtered HQ-50 subset.
Protocol and findings
Seven LLM agents were graded in isolated sandboxes with two independent proof checkers under three conditions: closed-book, frozen-retrieval, and live-web. Closed-book rediscovery is rare — at most 7 of 40 resolved problems — while live-web success mostly reflects fetching published solutions. Retrieval credit is fine, the authors argue, but it isn't the same evidence of discovery.
More from AGI Musings
- Follow-up: if speed is the crux, what's the optimal rate of revealing truths? — jessi_cata · 2026-11-02
- Debate with Robin Hanson: AI harms are mostly knowledge spreading, like libraries — jessi_cata · 2026-11-02
- David Patterson predicts job interviews, resumes and commutes all gone in 5 years — davidpattersonx · 2026-11-02
- New Yorker: AI Is Solving Decades-Old Math Problems, With Terence Tao as Guide — stevenstrogatz · 2026-11-02
- Dev argues we're underestimating cultural resistance to deep AI integration — andreisavu · 2026-11-02
- Ex-$500k/Year AI Engineer With 28 Years of Experience Says He's Now Homeless — ericelliott_ · 2026-11-02