27,280 coding-agent trials show the best harness changes with every model

shoumikgoswami · reddit · 2026-11-18

Runta and Baseten tested 22 coding-agent harness configurations across five models in 27,280 trials. Key finding: switching models changes which harness offers the best value — Codex CLI had the lowest cost per passed task with GLM-5.3 ($0.803) but ranked only 12th with Muse. A model upgrade is a reason to re-test your harness, not carry over the old setup.

Stability varies widely: on repeated resampling of their 62 tasks, Codex CLI + GLM-5.3 kept winning in 99.6% of samples, while the other four models' winners held their lead in only 25-53%.

Caveat: this is a vendor-run benchmark (Runta sells agent runtime infrastructure) — treat it as a prompt to test your own pairing, not a shopping list.

Original post →

More from coding & agent

coding & agent channel →