EurekaBench: GPT-6 Astra nearly matches humans on prediction but lags on scientific insights
geoffwolfe · x · 2026-10-04
ReasonCore unveiled EurekaBench, a new benchmark testing whether AI experiments can uncover mechanisms that explain observations, complementing SciCode (scientific code) and CritPt (research-level physics). It spans 26 problems and 306 insight questions across six domains, scoring scientific constraints, predictive accuracy, and mechanistic insights separately.
Key finding: GPT-6 Astra nearly matches the human reference on prediction (47.4% vs 48.8%) but trails sharply on insights (29.4% vs 69.7%) — fitting data isn't understanding it. ReasonCore is building a dataset around the benchmark.
Related event: EurekaBench: AI Excels at Problem-Solving but Lacks Scientific Insight(2 posts)→
More from Research
- Matt Green muses: are n^25 galactic algorithms normal, or a glimpse of provable Algorithmica? — matthew_d_green · 2026-11-20
- MIT and NVIDIA's QDA uses agents to auto-design LLM quantization recipes, beating top public ones by up to 2.2 points — omarsar0 · 2026-11-20
- AI-powered forensic metascience: auditing 7 million papers, 500K files scanned, 100+ issues found — gleech · 2026-11-20
- Harmonic's Aristotle writes proven-correct fusion plasma controllers in C and Verilog in 4 hours — satnam6502 · 2026-11-20
- Interp4discovery workshop on interpretability for scientific discovery at NeurIPS, Dec 12 — begusgasper · 2026-11-20
- Talk compares tabular foundation models TabPFN, TabICR vs gradient boosting — saheedniyi_02 · 2026-11-20