PyTorch Founder Admits Eval Harness Randomization Hurt Scores

Soumith Chintala acknowledged harness randomization hurt his model's coding eval scores and that it was never tuned for Codex; evaluator Josh Purtell explained he ran all effort levels with 3 rollouts each, taking the strongest.

2026-11-07 ~ 2026-11-07 · 3 related posts

• Eval method revealed: all effort settings, 3 rollouts each, scored on each model's strongest effort — JoshPurtell · 2026-11-07
• PyTorch creator Soumith Chintala admits eval harness randomization hurts their coding benchmark scores — soumithchintala · 2026-11-07
• Meta researcher: model never tuned for Codex, will improve harness randomization — JoshPurtell · 2026-11-07