Fixed seeds don't make scientific ML reliable: MAE swings 0.10–0.15 eV across splits
bravo_abad · x · 2026-11-05
Jorge Bravo Abad flags a common flaw in scientific ML papers: fixing randomstate=42 proves repeatability, not robustness. In his demo, band-gap prediction MAE swings from 0.10 to 0.15 eV (±0.02 eV SD) across five seed/split combinations on the same model and data.
Since many published algorithm comparisons differ by less than this variability (e.g. 0.11 vs 0.12 eV), such gaps don't establish superiority. His recommendation: report the spread across runs — mean ± SD over multiple seeds and splits — not just the best single number.
More from Research
- Adobe's Dramarrator turns books into editable audio dramas with a single prompt — amypavel · 2026-11-05
- Three robot posters and a world-model workshop talk at CoRL 2026 in Austin — breadli428 · 2026-11-05
- Typing Rules Become Programs in Tree Calculus, All Theorems Mechanized in Rocq — teropa · 2026-11-05
- Nature Communications study pinpoints high-level visual cortex as site of one-shot learning — MacrinePhD · 2026-11-05
- Ai2's StateBench: frontier models handle overwrites but drop to 13% on dependent state updates — ombre-ai · 2026-11-05
- MIT's UQ-DAAAM helps robots spot and fix unreliable object memories in real time — MacrinePhD · 2026-11-05