Same model, 9.2x token spread: harness tests show coding agent harness matters more than the model
zainhas · x · 2026-09-08
The author ran the same Three.js single-page generation prompt across different model+harness combos and found the harness affects token use and end-to-end time far more than the model itself.
Key findings:
- GLM 5.3 Flash Max varied wildly by harness: 475K tokens/9min on Codex vs 1.74M/30min on OMP vs 4.36M/20min on Opencode — a 9.2x token spread on one model.
- Qwen 3.8 27B x-high was fastest on Opencode (8m48s, 707K tokens) but slowest on OMP (41min, 3.4M tokens).
- The benchmark also logged TTFT, cache hit rates, tool calls/errors, and whether the output was actually opened in a browser and screenshot-checked.
Takeaway: harness choice, not model choice, dominates coding agent cost and speed — agent evals must treat the harness as a variable.
Related event: Coding Harness Matters More Than Model: Token Use Varies 9.2x(2 posts)→
More from coding & agent
- WanGP adds one-click AMD GPU support for Tencent's Hy-Nomas world model, runs on 8GB VRAM — cocktailpeanut · 2026-10-27
- He benchmarks new models by having them build their own copilot — $28 of compute — bucolucas · 2026-10-27
- Founder generates 46 app promo videos in 4 days with Claude, skipping $11,500 in UGC fees — PrajwalTomar_ · 2026-10-27
- Should agents be allowed to change plans mid-task? Reddit debates the drift boundary — wistfulcracker84 · 2026-10-27
- HeyGen + Canva MCP turns slide speaker notes into an avatar presentation video — HeyGen · 2026-10-27
- Local beats cloud: Qwen 3.8 FN handles private student data as well as frontier models — dr_lm · 2026-10-27