8 coding harnesses benchmarked on one model with fixed seed — results vary wildly even across runs
hideo_kuze_ · reddit · 2026-11-15
A Reddit post discusses a YouTube benchmark running the same model (Qwen 3.8 Flash Next, Unsloth UD-IQ4XS) on an RTX 5090 + 128GB DDR5, building a pixel-art "Island Fishing" game across OpenCode, DeepSeek Harness, Pi, Cline, Hermes Agent, Kilo Code, Codex CLI, and VSCodium+Continue — fixed seed, temperature 0, 3 attempts each.
Key points:
- Even the same harness produced very different results across its 3 attempts, showing high test noise.
- The poster flags a design flaw: no clarifying questions allowed and no specs for view perspective or pixel size, which wildly affect outcomes.
- Fastest harnesses: kilo, opencode, hermes; 2-3x slower: pi, codex, cline, deepseek harness.
- Best results: opencode, pi, codex.
The author asks whether others agree with the methodology and which harnesses people prefer per task.
More from coding & agent
- Dev reports Unreal's built-in AI assistant closes the editor for rebuilds with no way to restart it — FanaHOVA · 2026-11-15
- Claude deletes professor's year of lecture notes while asked to renumber pages — petrusenko_max · 2026-11-15
- Feed vintage synth schematics to Claude and get working instrument emulations in 2-30 minutes — churchkey · 2026-11-15
- CrowdGPT pitches a community-driven, datacenter-less LLM — prove Altman wrong about personal DRAM — Vxtzq1 · 2026-11-15
- Dev argues headless browsing is a stopgap; agents should prefer MCP/CLI/API paths — mobileraj · 2026-11-15
- Kevin Roose's "council of Claudes" caught book errors human editors missed — every · 2026-11-15