Custom Agent Harness Test: GLM-5.2 Best Overall, DeepSeek Flash Top Value
codes_astro · reddit · 2026-08-14
A developer built a custom agentic harness using the Pydantic Agent framework to test recent open-weight models (like GLM-5.2, Kimi-K3, MiniMax M3, and DeepSeek V4 variants) on real coding tasks.
The setup uses a build → review → fix loop without external judge models, tracking tokens, cost, runtime, and repairs. Key findings:
- GLM-5.2: Best overall quality pick with strong UI and first-pass accuracy, needing only 1 repair across 9 runs.
- MiniMax M3: Best for speed and cost, producing usable outputs quickly and cheaply.
- DeepSeek V4 Flash: Surprise value model, offering the lowest cost while outperforming V4 Pro in practical usability and app logic.
- Kimi-K3: Despite strong public benchmarks, it was the most repair-heavy and inconsistent in this specific harness.
The main takeaway is that public benchmarks do not fully predict real-world agent performance, and usable output per loop is a better signal.
More from coding & agent
- AgenticROS: Building a Cloud Control Plane for Physical AI and Robots — chrismatthieu · 2026-08-14
- Arcee Team Shares Multi-Agent Coding Workflow for Complex Tasks — code_star · 2026-08-14
- Context Windows Are Not Memory: A Guide to AI Agent Memory Architecture — blaizedsouza · 2026-08-14
- Production AI Agent Security: Just-in-Time Permission Elevation Framework — blaizedsouza · 2026-08-14
- 20-Year Engine Veteran Builds SF Bay Area Digital Twin from Scratch Using AI — julianharris · 2026-08-14
- Agent Arena Leaderboard: Claude Opus 5 Leads, Kimi K3 Hits Top 5 — arena · 2026-08-14