The industry "solved" benchmark gaming by simply stopping making evals
JoshPurtell · x · 2026-11-02
Josh Purtell observes that in the past there were excellent purpose-built evals, and the n-1 other labs would train on them, benchmaxx, and get duly criticized for it. Now he suspects the problem was "solved" — by simply not making evals anymore.
This ties into his earlier point: the best evals are necessarily Goodhart-able, and the industry seems to have given up on designing independent evals for that reason.
More from Research
- J-Lens is just a first-order Taylor expansion, researcher points out — burny_tech · 2026-11-02
- Mollick: expert-level work benchmarks are brutally hard, and ARC-AGI, GDPval, METR are saturated — emollick · 2026-11-02
- Rebuilding a Protein Domain Distribution Pipeline After a Backup Failure — AryHHAry · 2026-11-02
- NVIDIA's Elastic Latent Recurrence lets one 4B model trade loop count for accuracy: 18→29 on AA Index — burny_tech · 2026-11-02
- IBM's STAIR retriever cuts invalid references 65× and hits 82.6% accuracy by using document structure — solyarisoftware · 2026-11-02
- ReLift Hits SOTA Dense 3D Tracking on WorldTrack With 2.7x Faster Pipeline — Technion · 2026-11-02