Re-running the same agent task flips results: minimal harnesses are most stochastic
yb2698 · x · 2026-10-14
The author examined harness 'stochasticity': what happens if you re-run the exact same task N times with the same model and harness? Using TB-Lite tasks already run 3 times, they measured consistency across pass/fail flips, turn-count variation, and input-token variance. Minimal harnesses (ipython, pi) suffer most, while OpenHands is by far the most consistent — which the author attributes to its implementation after analyzing trajectories.
More from coding & agent
- Creator builds custom GoPro ingestion tool with Codex: 4-camera array with WiFi trickle ingest — OpenAIDevs · 2026-10-15
- Claude Code 2.1.300 cuts system prompt tokens by 58.5%, drops forced question-asking directive — ClaudeCodeLog · 2026-10-15
- Claude Code 2.1.300 ships 88 CLI changes, drops AskUserQuestion directive from system prompt — ClaudeCodeLog · 2026-10-15
- Tahoe AI opens Tara, a one-prompt plug-in connecting agents to its Tahoe-100M biology datasets — anshulkundaje · 2026-10-15
- sgrep cuts coding agent LLM costs 21.3% on SWE-bench versus ripgrep, fully local in Rust — udmrzn · 2026-10-15
- 10 GitHub repos that turn Claude Code and Codex into elite UI designers — udmrzn · 2026-10-15