Stanford researcher flags data leakage and confounding in BioReason benchmark

Stanford's Anshul Kundaje publicly questioned the BioReason benchmark, citing evidence that shuffling input sequences barely changes model outputs, along with prompt-level data leakage and confounded positive/negative sample selection.

2026-11-03 ~ 2026-11-03 · 2 related posts

• Researcher flags BioReason benchmark flaws: model unresponsive to randomized inputs — anshulkundaje · 2026-11-03
• BioReason benchmarks 'seriously flawed': prompt leakage and set-selection confounds alleged — anshulkundaje · 2026-11-03