Stanford's Kundaje blasts AI×Bio research rigor and NeurIPS reviewing
On October 3, Stanford professor Anshul Kundaje (computational biology / deep learning) posted a series of critiques on research norms and top-conference reviewing in AI×Bio, naming serious problems in the BioReason paper and its benchmark, and offering concrete advice for AI researchers moving into biology.
Confirmed
- Kundaje said the BioReason benchmark has serious task-definition flaws and prompt-level information leakage, and recommended no longer using these tasks; that it ranked in the year's NeurIPS top 20 reflects, in his view, the poor reviewing of biology work at ML conferences — "a complete joke."
- He clarified his criticism targets the BioReason paper, not BioReason-Pro; students in his class found BioReason-Pro may have other issues, pending verification.
- He argued the claimed "multimodal biological reasoning model" is mislabeled and lacks basic controls; early half-baked research making hype claims without controlled experiments is a worrying AI-for-Science trend.
- He said many AI practitioners ignore controls entirely, entertain magical thinking like "batch effects will be emergently fixed by the model," and that expecting a transformer trained on billions of cells to magically learn causality reflects absent first-principles thinking.
- He attributed the problem to this generation of "engineering-style" AI practitioners rather than CS itself: pre-deep-learning AI/ML scholars had strong causal-inference grounding, while a generation of talented "model engineers" treats every problem as hill climbing — ineffective in biology where the "hill" is hard to define and confounders abound.
- He noted that when models are in the loop or data is generated for specific learning tasks, control-design rules differ from classical experiments.
- His advice for newcomers: work with wet-lab scientists and systematically study experimental design, confounders, and controls.
- He called for rigor to increase 5x and hype to decrease 100x.
Unconfirmed
- Whether BioReason-Pro has other issues awaits verification; only a preliminary check by his students exists so far.
Why it matters
- A prominent scholar spanning computational biology and deep learning directly criticized AI4Science methods, touching benchmark quality, missing controls, and conference reviewing — with his point on model-in-the-loop experiment design offering direct guidance for AI×Bio researchers.
2026-10-03 ~ 2026-10-03 · 14 related posts
Primary sources
- [source] Researcher slams NeurIPS reviews as BioReason top-20 pick called a farce — anshulkundaje · 2026-10-03
- [source] Stanford prof warns BioReason benchmark has task definition flaws and prompt leakage — anshulkundaje · 2026-10-03
- Stanford's Kundaje Flags BioReason Paper for Missing Basic Controls Amid Hype — anshulkundaje · 2026-10-03
- Stanford's Kundaje Slams 'Hypey' AI4Science Studies Published Without Basic Controls — anshulkundaje · 2026-10-03
- Stanford professor slams AI×Bio hype: half-baked studies lacking basic controls — anshulkundaje · 2026-10-03
- [source] Stanford professor to AI folks entering bio: learn CONTROLS from wet-lab scientists — anshulkundaje · 2026-10-03
- Stanford's Anshul Kundaje: AI-bio newcomers must learn experimental design and controls — anshulkundaje · 2026-10-03
- Stanford professor Anshul Kundaje calls out sloppy rigor in AI x Bio research — anshulkundaje · 2026-10-03
- Stanford's Kundaje: model-in-the-loop biology needs different experimental controls — anshulkundaje · 2026-10-03
- Stanford Professor Slams AI-for-Biology Work That Ignores Controls and Causality — anshulkundaje · 2026-10-03
- Stanford Professor: Don't Expect Causality to Emerge from Training on Billions of Cells — anshulkundaje · 2026-10-03
- Stanford prof: the problem is a generation of AI 'engineers', not computer scientists — anshulkundaje · 2026-10-03
- Stanford prof: hill-climbing model engineers don't work in biology — anshulkundaje · 2026-10-03
1 near-duplicate retellings: ShenRaphael