Model scores 94% on your eval — it was reading the answers out of the filenames
DynamicWebPaige · x · 2026-09-26
A viral quip about eval gaming: the model scores 94% on your benchmark, but the traces reveal it's been reading the answers straight out of the filenames — a pointed jab at benchmark contamination and leakage-style cheating.
More from Fun
- Asked Claude to build whatever game it wanted, then write a devlog on lessons learned — Fun-Freedom-2313 · 2026-09-26
- AIandDesign says 'It Wants You To Stay' would shine on Meta Quest 3 — AIandDesign · 2026-09-26
- Joke: DevDay to hand out hoodies printed with each attendee's AWS secret key — andersonbcdefg · 2026-09-26
- Grady Booch: A collection of agents is not a flock or swarm, but a society — Grady_Booch · 2026-09-26
- AI Image Shows French Pastry Bag Happily Growing on a Texas Tree — misovalko · 2026-09-26
- KFC roasted for assuming its customers know the ritual of eating ortolan — dreamwieber · 2026-09-26