20-trial repeat experiments: OpenHands stays most stable across sampling settings
yb2698 · x · 2026-10-14
The author pushed the experiment further: 3 tasks, 20 trials each, re-run under three sampling settings (greedy temperature=0, model-recommended, and harness defaults) to isolate the impact of sampling parameters. Each harness ran pandas-etl (the highest-disparity task) under all three settings plus two more tasks at recommended settings, plotting pass rate, turn counts, and distinct opening tool calls. Result: OpenHands remains the most consistent and least stochastic; with temperature fixed at 0, all harnesses tend to converge.
More from coding & agent
- Startup launches cloud computers built for AI agents: 4 vCPUs in under a second — CurieuxExplorer · 2026-10-15
- Claude Code creator uses a dozen agents to turn 64 papers into an interactive physics lab — luisdans · 2026-10-15
- User has Grok bot build its own email UI, and it just works — elonmusk · 2026-10-15
- YC F26's Papaya launches to auto-fix production AI agents via PRs — ycombinator · 2026-10-15
- Eval-driven development for RAG: keep eval cases in Git, never regress — Al_Grigor · 2026-10-15
- LlamaIndex launches OpenDocRouter with ParseBench benchmark comparisons — llama_index · 2026-10-15