Meta researcher: model never tuned for Codex, will improve harness randomization
JoshPurtell · x · 2026-11-07
In a discussion about evaluating pi, CC, and swe-mini, a user noted "our model is never tuned for Codex — any other harness works much better with it." Meta's Soumith Chintala responded that they will do better harness randomization to cover Codex, admitting they still have a long way to go to generalize like frontier models do. Josh Purtell said rollouts for pi, CC, and swe-mini are running now.
Related event: PyTorch Founder Admits Eval Harness Randomization Hurt Scores(3 posts)→
More from coding & agent
- Trail of Bits: Three Lessons From Running Hundreds of AI Agents on Critical OSS — banteg · 2026-11-07
- Dev builds a sailing sim with Claude Code and pits frontier models against each other — abhi3188 · 2026-11-07
- Theory Ventures hosts AI agent hackathon: 5 hours to run the most profitable bakery in town — HamelHusain · 2026-11-07
- 36K job postings: Claude Code passes Cursor, a third of 'agentic' roles are buzzword — vilnitskiy · 2026-11-07
- Theo on T3 Code positioning: know exactly who your product is actually for — IsForAt · 2026-11-07
- Google study: devs expect agents to perform as-is while juniors grow via review — rseroter · 2026-11-07