Custom Agent Harness Test: GLM-5.2 Best Overall, DeepSeek Flash Top Value
codes_astro · reddit · 2026-08-14
A developer built a custom agentic harness using the Pydantic Agent framework to test recent open-weight models (like GLM-5.2, Kimi-K3, MiniMax M3, and DeepSeek V4 variants) on real coding tasks.
The setup uses a build → review → fix loop without external judge models, tracking tokens, cost, runtime, and repairs. Key findings:
- GLM-5.2: Best overall quality pick with strong UI and first-pass accuracy, needing only 1 repair across 9 runs.
- MiniMax M3: Best for speed and cost, producing usable outputs quickly and cheaply.
- DeepSeek V4 Flash: Surprise value model, offering the lowest cost while outperforming V4 Pro in practical usability and app logic.
- Kimi-K3: Despite strong public benchmarks, it was the most repair-heavy and inconsistent in this specific harness.
The main takeaway is that public benchmarks do not fully predict real-world agent performance, and usable output per loop is a better signal.
More from coding & agent
- Dev builds MCP server that lets AI agents outsource phone calls to real humans — Many_Benefit_3393 · 2026-10-16
- A vector DB is not LLM memory: long thread maps write/read paths, failure modes and evals — techNmak · 2026-10-16
- Mergelock: open-source MCP server catches semantic merge conflicts git misses across parallel AI agents — ksadayamuthu · 2026-10-16
- OpenUI 1.0 launches open source, turning agent text answers into live interactive UIs — heyshrutimishra · 2026-10-16
- 10 GitHub repos to see what your AI agent really did, from Langfuse to claude-code-log — victor_explore · 2026-10-16
- Mosaic: multi-session agentic RL with hierarchical advantage estimation — LARK-Lab · 2026-10-16