Stanford reran 50 famous LLM findings on 20 models — 44% no longer hold
heyshrutimishra · x · 2026-11-04
Stanford's Owen Queen and James Zou reran 50 widely cited LLM behavior claims on 20 models from 8 labs released July 2024–Sep 2026: 22 expired, 17 weakened, 8 persist, 3 strengthened — 78% have faded or weakened. Expired claims (median lifespan 14.7 months) are mostly classic capability failures: letter counting, non-decimal arithmetic, lost-in-the-middle, hallucinated citations, caving under pushback. What persists or strengthened: sycophancy, self-preference in LLM judges, and non-random "randomness." Capability problems get fixed fast; behavior problems stick around.
Related event: Stanford Retests 50 LLM Findings: Nearly Half Fail Within 15 Months(8 posts)→
More from Models
- Leaker hints: no "Fable 5.5" or "GPT-6.2 Astra" today — but Thursday is the date — daniel_mac8 · 2026-11-04
- GPT-6.2 spotted: New OpenAI model rumors point to "astra" incoming — udmrzn · 2026-11-04
- Larger tokenizers suffer more multi-epoch regret, small vocabs survive repeats — SonglinYang4 · 2026-11-04
- Anime image forensics tool foidfinder detects character mashups and booru tags — AlpinDale · 2026-11-04
- Sonnet reportedly behaves with a different personality on Medium vs Max tiers — AaronBergman18 · 2026-11-04
- Mystery Stealth Model Crouton Alpha Free for a Week With 1M Context; Guessed to Be Meta's Wonder Successor — imjustnewatai · 2026-11-04