Benchmark: pi agent harness outperforms DeepSeek's dsh on cost
solyarisoftware · x · 2026-08-20
DeepAPI benchmarked DeepSeek's official agent harness (dsh) against the pi framework.
- Test Scale: 180 controlled runs across 3 models and identical tasks.
- Cost: dsh was more expensive than pi on 2 out of 3 models.
- Analysis: The difference stemmed from round-trips; dsh required 11 model calls versus pi's 8.5 (on DeepSeek V4 Pro).
- Surprise: The only model where dsh won was Kimi K3, not DeepSeek.
Note: DeepSeek V4 Pro was run on GMICloud fp8, not the official API.
More from coding & agent
- Apple code signing drags for hours; official advice: just wait a week — pjausovec · 2026-10-16
- Hermes: a 4B model with 8K context got worse with more test-time compute, pointing to contextual reasoning — mangahomanga · 2026-10-16
- Flux 3 Klein's Default Templates Are Broken Without Its Prompt Upscaler — StrictMove5839 · 2026-10-16
- 21-person team ships hundreds of PRs weekly with AI writing 97% of the code — rseroter · 2026-10-16
- ClosedMCP: Set Up MCP Servers Once, Share Them Across Your Whole Team — Ok_Sprinkles1301 · 2026-10-16
- Heisentools: let an agent rewrite a mock.py until it mirrors any real tool's API — kieranklaassen · 2026-10-16