AI×Bio benchmarks called out as LLM-generated "benchslop" that may be impairing bio-reasoning

anshulkundaje · x · 2026-11-13

@EganPeltan argues frontier models keep making simple reasoning mistakes in biology, likely because the literature and benchmarks are polluted. Many AI×Bio benchmarks, he says, follow a loop of LLM-generated questions → "expert" review → LLM benchmarked → LLM graded → LLM summarized — and benchmaxxing on this benchslop may be degrading bio-reasoning. Good biological benchmarks matter for training frontier models, and details are critical: one base, one atom, or one stereocenter can separate a nasal decongestant from a tooth-rotting psychotic stimulant.

Original post →

More from Models

Models channel →