Researcher flags BioReason benchmark flaws: model unresponsive to randomized inputs
anshulkundaje · x · 2026-11-03
Anshul Kundaje publicly challenged claims that BioReason was 'successfully used', citing an independent evaluation showing the model barely responds when input sequences are randomized. He adds the benchmarks are seriously flawed, with prompt leakage and likely confounds in positive/negative set selection even after fixes by the Hugging Face team.
More from Models
- MIT: AI edits 0.016% of another model's weights, lifting MATH-500 from 7.9% to 48.1% — rohanpaul_ai · 2026-11-03
- Schmidhuber: AI weight editing is 'old hat' from his 1990s fast weight programmers — SchmidhuberAI · 2026-11-03
- Debate over 'GPT-6 edits Qwen weights': critic says it's just direction ablation, not weight editing — IanArawjo · 2026-11-03
- NorthCUA-Pro hits 93.3 on ScreenSpot-Pro, edging out GPT-6 Astra for new SOTA — NielsRogge · 2026-11-03
- Mystery 'Pulse Alpha Free' model with 1M context appears stealth-free on OpenCode — mark_k · 2026-11-03
- Anthropic blocks Claude Code from cloning existing video games after knockoff wave — The Decoder · 2026-11-03