Stanford prof: clever shortcut solutions exploiting eval structure keep winning AI contests
anshulkundaje · x · 2026-11-20
Stanford professor Anshul Kundaje wraps up the cell-simulation contest debate: "The fact that this type of solution won the contest only further reinforces what we've observed countless times in challenges — clever approaches that exploit the structure of the data & the evals often win." A pointed reminder that contest winners may exploit benchmark loopholes rather than truly generalize.
More from Research
- MIT's Hiroshi Ishii shares Tangible Media Group's Creative Tension principles — ishii_mit · 2026-11-20
- Train a real LLM entirely in your browser with this interactive demo — morgymcg · 2026-11-20
- Ex-OpenAI Researcher: English Is the New ML Programming Language, Lean the Type System — lukaszkaiser · 2026-11-20
- S1CogBench probes System One decision models across 24 cognitive tasks — Dengzhe Hou · 2026-11-20
- AFFECT from KAIST makes world models dynamics-faithful via action effect matching — John Won · 2026-11-20
- Mind2Act World: a 180-task benchmark exposing robots' mind-action composition failures — Mind2Act-World · 2026-11-20