A good eval has a shelf life of maybe 3 months, practitioner argues
pvncher · x · 2026-09-23
Practitioner pvncher argues a good eval stays fresh for maybe 3 months, 6 at best. If you're relying on an older eval to judge whether a model is good, it's probably lower-signal than asking a random person's opinion—the point being benchmarks go stale and get gamed as models iterate, so evals need constant refreshing.
More from coding & agent
- Can Jev Write Graphical Models? LLM Meets Factor Graphs and Bayesian Nets — blaizedsouza · 2026-09-23
- Jev explained: the semantic decision layer between rules and LLMs — blaizedsouza · 2026-09-23
- A complete AI workflow for turning ideas into product launch videos, prompts included — Aiden_Tech_Ai · 2026-09-23
- A 19-year-old built a Claude Code trading bot in 2 days, claims $750K profit — Aiden_Tech_Ai · 2026-09-23
- Dev sends coding tasks from 35,000 feet with Codex Remote: "this will never cease to amaze me" — Dimillian · 2026-09-23
- PM vibes coding at a big tech firm: 1-2 months of work now done in a week — vista8 · 2026-09-23