Meme: eval guy keeps pulling fresh benchmarks from Artificial Analysis as models saturate them
jd_pressman · x · 2026-09-04
An AI-circle joke: an 'eval guy' complains models keep saturating his benchmarks, and when asked where he gets more, admits he just grabs new ones from Artificial Analysis—effectively feeding benchmarks to models. The punchline features François Chollet jumping in, poking fun at benchmark saturation and contamination.
Related event: AI Community Jokes That Every Benchmark Will Eventually Be Saturated(2 posts)→
More from Fun
- Codex Down for Much of Rollout Day — and Users Not Getting New Model Either — RexDouglass · 2026-09-04
- Security researcher mocks over-tightened LLM cyber safeguards — moyix · 2026-09-04
- Non-programmer recreates a DOOM-style game in hours with OpenAI's Astra, nearly bug-free — DeryaTR_ · 2026-09-04
- How do you cope with new model release anxiety? You can't — generativist · 2026-09-04
- 'AI screwed up and now everyone's gonna be homeless' meme spreads in AI circles — bennash · 2026-09-04
- Is Astra AGI? Five contradictory answers that are all true — sandersted · 2026-09-04