Eval method revealed: all effort settings, 3 rollouts each, scored on each model's strongest effort
JoshPurtell · x · 2026-11-07
Replying to Soumith Chintala's question about why 'low' effort is used for inkling 2 small, Josh Purtell explains the methodology: every effort setting a model provides is run, with 3 rollouts per effort per task, and scores are taken from the model's strongest effort setting.
The rationale: frontier models differ — some perform better at high than max, others peak at max — so running all settings and taking the best avoids misjudging model capability due to effort-level configuration.
Related event: PyTorch Founder Admits Eval Harness Randomization Hurt Scores(3 posts)→
More from Models
- Mistral's $5.99 student plan bundles $165 worth of DeepSeek and GLM usage — GuillaumeLample · 2026-11-07
- 11,398 real-world robot evals across 124 sites confirm GPT-6 Astra is seriously strong — DJiafei · 2026-11-07
- Google DeepMind's Ivan Leo to reveal how agents cheat on benchmarks at scale at AIE Code — jocarrasqueira · 2026-11-07
- OpenAI launches BioBasicsBench: 100 real wet-lab tasks to stress-test AI agents — OpenAI · 2026-11-07
- SoMark's 1.3B OXR tops olmOCR-Bench, OmniDocBench with pluggable layout module — pmttyji · 2026-11-07
- Tomo launches Dachi 1.0, a companion-focused model beating Opus on retention — Jackyhuang · 2026-11-07