DeepSeek Harness Eval Bug Triggers Agent Infinite Loop, Tripling Cost
teortaxesTex · x · 2026-08-14
While evaluating DeepSeek Harness (dsh), BohuTANG discovered a critical agent policy bug: when grep returns an exit code 1 for zero matches, dsh treats it as a failure requiring investigation. This causes the agent to continuously create and fix additional verification steps even after tests have passed, trapping it in an infinite loop.
Under identical task, model, and reasoning configurations, this bug leads to massive resource waste:
- Requests: 61 vs normal 32
- Duration: 10m38s vs 4m55s
- Cost: $0.17 vs $0.05
- Input tokens: 2.9M vs 548K
More from coding & agent
- Terminal Night 3 talk slides released, missing demo video to follow — 4310sy · 2026-10-16
- Agent developers are sitting on unused API credits that can exceed actual usage — EnvironmentalDay8864 · 2026-10-16
- Notion's bet on agents: abstract away deployment so users and agents just write code — simonguozirui · 2026-10-16
- Four Gemini 4 Argon agents beat Minecraft's Ender Dragon autonomously in 27 hours — Maxence Faldor · 2026-10-16
- Tencent's Terminal-RSI builds recursively self-improving terminal agents, hitting 19.84% on Terminal-Bench 4.0 — tencent · 2026-10-16
- Tsinghua's AutoSFT uses multi-agent collaboration to automate SFT data research — TsinghuaNLP · 2026-10-16