"There is no science of transformers" — researcher slams irreproducible AI research
gerardsans · x · 2026-11-19
In a pointed exchange with Alex G. Dimakis, gerardsans argues there is currently "no science of transformers": AI research is done poorly, with no real agents — just different inference runs. With hidden variables everywhere, systems can't be studied or diagnosed; no null hypothesis means no conclusions are possible.
He adds that running inference and reporting results is a wasteful way to describe a setup, since no reproducible configuration can be isolated — "This is a survey. Not science."
More from Research
- The Little Book of ML Metrics nears launch, pre-order at $60 for data scientists — santiviquez · 2026-11-19
- Schmidhuber publishes annotated AI history tracing deep learning to 1676 — SchmidhuberAI · 2026-11-19
- ICLR debate: cap author rebuttals — and reviewers' 32 AI-generated weaknesses too? — TuhinChakr · 2026-11-19
- Weight-space blend of base and RL checkpoints restores the diversity RL removed: α=0.25 keeps 64-80% of pass@1 gain — Perfect_Power815 · 2026-11-19
- Erik Hoel: consciousness is a great mystery, but its definition isn't — experts largely agree — koltregaskes · 2026-11-19
- ICLR author expands rebuttal from 24 to 40 pages, gets zero reviewer replies — O_Fake_One · 2026-11-19