PyTorch creator Soumith Chintala admits eval harness randomization hurts their coding benchmark scores
soumithchintala · x · 2026-11-07
Soumith Chintala (PyTorch) responded to Josh Purtell's coding eval results, saying the team will improve harness randomization to better cover Codex, suspecting it explains part of their underperformance.
Purtell explained the methodology: for each task they run all effort settings provided by the model with 3 rollouts per effort, scoring at each model's strongest setting — since some frontier models do better at high vs max. Chintala acknowledged they have "a long way to go" to generalize to frontier-model level and thanked him for the signal.
Related event: PyTorch Founder Admits Eval Harness Randomization Hurt Scores(3 posts)→
More from coding & agent
- Claude hands user a Lovecraftian Bash script then says 'things to look for are…' — teodorio · 2026-11-07
- iOS 27.3 reveals Siri's memory graph: memory atoms fed by Fastpass and agent recipes — alexisgallagher · 2026-11-07
- Indie dev adds AI agent contact editing with per-change undo and approval gates — SeanOliver · 2026-11-07
- Designing an MCP server for 60+ financial datasets: entity keys, per-connector auth, context budgets — vianexus_official · 2026-11-07
- zeeg to livestream Junior in action, breaking down technical details of the agent — zeeg · 2026-11-07
- Sketching musical interfaces with ASCII art using Gemini in Google Antigravity — alexanderchen · 2026-11-07