Re-testing 50 cited LLM findings: most fail within 15 months
AaronBergman18 · x · 2026-11-03
James Zou's team re-tested 50 widely cited findings about LLM behavior (biases, hallucination, etc.): within 15 months, most no longer held or had substantially reduced effects—LLM findings have a short shelf life. Quoter Dylan Matt adds that in AI, a "peer-reviewed" label makes a finding less trustworthy, since peer review lags the field badly.
Related event: Stanford Retests 50 LLM Findings: Most Fade Within 15 Months(3 posts)→
More from AGI Musings
- Giraffe evolution meme: AI's low-hanging fruit keeps hanging higher every year — itsOmSarraf_ · 2026-11-03
- Bain's $6T vs Goldman's $0.5T: How Big Can AI Revenue Really Get — luisdans · 2026-11-03
- Databricks: Agents now run AI research loops, training jobs grew 8x in a year — matei_zaharia · 2026-11-03
- AI elites might hoard intelligence like mercantilists hoarded gold, argues Venkatesh Rao — The Decoder · 2026-11-03
- Same AI, cents vs tens of millions: the value gap is in the question, not the answer — r0ck3t23 · 2026-11-03
- AI researchers: one success in an environment loop is enough signal to run RL on it — cephaloform · 2026-11-03