Screening 307 AI-for-science papers in one week: the test that matters is out-of-distribution
bravo_abad · x · 2026-10-28
The latest Discovery at Scale issue catalogued 307 peer-reviewed AI-for-science articles from Oct 21–27 across physics, chemistry, earth science, biology and medicine (14 read closely, 27 cited), asking one question: does the model still work on data it has never seen?
Several of the week's most useful papers test exactly that: a materials group held out every fluorine compound and found uncertainty-driven searches did no better than random picking; a glacier model was scored on unseen glaciers; a single-cell model trained on wild-type B cells predicted a mutant, validated by mouse crosses.
Key takeaway: most scientific ML papers report numbers on data resembling the training set; the decisive design choice is where to put the test data. If you're building a model, first decide which shift your scientific claim depends on, then make that shift the test.
More from Research
- Eric Topol highlights immunome paper combining multi-assay data with AI to model immune aging — EricTopol · 2026-10-28
- Google Research's 'sleep framework' lets LLMs keep learning, retaining up to 10M tokens — behrouz_ali · 2026-10-28
- CATCH: ~200 examples of adversarial training make LLMs robustly honest across benchmarks — irinarish · 2026-10-28
- Andrew Critch's dubious credences: P≠NP at 80%, Riemann hypothesis at just 60% — AndrewCritchPhD · 2026-10-28
- AI-driven Indian biotech PopVax, backed by $20M from Vitalik's Balvi fund, enters human trials with fridge-stable broad-spectrum Covid vaccine — chaitjo · 2026-10-28
- Cognition launches FrontierCode 2.0, a coding benchmark testing judgment beyond the spec — DevinAI · 2026-10-28