STEMBench launches: 6 STEM domains, zero full-target successes across all runs
mariofilhoml · x · 2026-10-15
Ulam soft-launched STEMBench, a correctness-first evaluation across six STEM domains (biology, chemistry, physics, astronomy, Earth science, engineering), with Hy4 Preview included. Each run: 120 tasks (20/domain), 10 runs, 1,200 scored responses.
Scoring uses a 0-4 fractional progress scale, with closed-book vs tool-enabled settings. Notably, zero full-target successes occurred across all ten runs—best scores sit in the 2.17-2.99 'meaningful partial' range, underscoring how far models remain from genuine research-level reasoning. Public test set and ErdosBench are available.
More from Research
- Pathway CSO pitches BDH as post-Transformer architecture uniting memory, reasoning, adaptation — schwentker · 2026-10-15
- EmbodiedEvo tops RoboDojo at 38.95% success by letting robots inherit failed practice — thetripathi58 · 2026-10-15
- 797GB of n-gram counts in CPU RAM let 1B models match 57% more compute — PangWeiKoh · 2026-10-15
- Google's HARNESS² evolves agent harnesses without verifiers, up to +11.5 points on WorkBuddy-Bench — dair_ai · 2026-10-15
- Subbarao Kambhampati to dissect the 'mythos of LLM thinking tokens' at UMN — rao2z · 2026-10-15
- How 3D Gaussian Splatting gives robots spatial memory of what's out of view — TheTuringPost · 2026-10-15