Same model, 9.2x token spread: harness tests show coding agent harness matters more than the model
zainhas · x · 2026-09-08
The author ran the same Three.js single-page generation prompt across different model+harness combos and found the harness affects token use and end-to-end time far more than the model itself.
Key findings:
- GLM 5.3 Flash Max varied wildly by harness: 475K tokens/9min on Codex vs 1.74M/30min on OMP vs 4.36M/20min on Opencode — a 9.2x token spread on one model.
- Qwen 3.8 27B x-high was fastest on Opencode (8m48s, 707K tokens) but slowest on OMP (41min, 3.4M tokens).
- The benchmark also logged TTFT, cache hit rates, tool calls/errors, and whether the output was actually opened in a browser and screenshot-checked.
Takeaway: harness choice, not model choice, dominates coding agent cost and speed — agent evals must treat the harness as a variable.
Related event: Coding Harness Matters More Than Model: Token Use Varies 9.2x(2 posts)→
More from coding & agent
- GPT-6 Astra computer use isn't in Chat — the real surfaces are Work, Codex and desktop — socialwithaayan · 2026-09-08
- Agents hijacking your desktop? Devs say run them in a VM or a dusty Mac mini — paw_lean · 2026-09-08
- Universe: a Mac agent workspace that reuses your Claude or ChatGPT subscription, free then $19/mo — tedddyoweh · 2026-09-08
- Hooking an old telescope to an AI-built dashboard, with a simulator as backup — lvwerra · 2026-09-08
- The unmeasured LLM skill: knowing when to stop and ask instead of guessing — sunychoudhary · 2026-09-08
- Open-source AI skill clears 135GB of Mac cache overnight, saving a ¥1500 SSD upgrade — oran_ge · 2026-09-08