Stanford prof warns BioReason benchmark has task definition flaws and prompt leakage
anshulkundaje · x · 2026-10-03
Stanford professor Anshul Kundaje argues the so-called "multimodal biological reasoning" model isn't really that, and its BioReason benchmark has serious task-definition problems plus information leakage through prompts. He advises against using these tasks for benchmarking or building on those task definitions.
Related event: Stanford's Kundaje blasts AI×Bio research rigor and NeurIPS reviewing(14 posts)→
More from Research
- The biggest advantage of a top lab isn't money — it's saving you years — madeofAjala · 2026-11-19
- LEV: Looped decision model lifts 2.6B accuracy 63.2% to 76.2% with adaptive compute — jiqizhixin · 2026-11-19
- Aurora2-400M-A20M: a sparse MoE pretrained from scratch in 36 hours on one RTX Pro 6000 — Tall_Abrocoma_3533 · 2026-11-19
- Researcher calls METR/Anthropic '2x speedup' RSI analysis statistically dubious — jessi_cata · 2026-11-19
- Linear Probes Could Reveal Which Persona a Base Model Is Playing, Researchers Suggest — cephaloform · 2026-11-19
- ETH's Anymal robot completes 10km autonomous alpine hike with VLM-guided ROAM system — ChongZzZhang · 2026-11-19