Chollet Warns Against Benchmark Cramming as GSM1k Rebuild Exposes Contamination

François Chollet argues that, like evolution, models should optimize generalization and robustness rather than benchmark cramming. A GSM1k rebuild of GSM8K confirms contamination, with some models dropping 8 points, prompting calls to evaluate enterprise agents on real-world data.

2026-11-02 ~ 2026-11-02 · 2 related posts

• François Chollet: Evolution doesn't benchmaxx — it optimizes for generality and robustness — fchollet · 2026-11-02
• GSM1k Rebuild Exposes Benchmark Contamination: Up to 8-Point Drops, Agent Eval Advice — krishnan · 2026-11-02