Re-testing 50 cited LLM findings: most fade within 15 months
cephaloform · x · 2026-11-04
James Zou's team re-tested 50 widely cited findings about LLM behavior (biases, hallucination, etc.). Within 15 months, most no longer held or had substantially reduced effects — LLM findings have a short shelf life. A comment notes papers claiming claims about "language models" as a class are really about the few specific models available at the time.
Related event: Stanford Retests 50 LLM Findings: 44% Dead Within 15 Months(7 posts)→
More from Research
- Perplexity and turbopuffer build SOTA contextual embedding model for retrieval — JensHonack · 2026-11-04
- Native image reading beats OCR for deep research agents, cutting tool calls — RulinShao · 2026-11-04
- CMU's IdeaTaste benchmark: LLM research taste reaches only ~26% of human level — CMU-SCS · 2026-11-04
- JFPuget Team Takes 3rd on Public Leaderboard of Both Kaggle ARC-AGI Competitions — JFPuget · 2026-11-04
- Stanford Data-Centric AI Workshop 2026 kicks off with full-day live talks — jure · 2026-11-04
- New paper: benchmarks are surprisingly compaction-friendly, no method beats self-summarization — khademinori · 2026-11-04