Agent Test: Switching Models Cuts 70% of Costs

qubridInc · reddit · 2026-07-11

The author shared an internal research agent test: a single task typically requires 40–60 LLM calls, and the cost per task exceeded $1 when using GPT-4o.

After switching most "routing/extraction" calls to open-source models, they ran the same evaluation suite (about 300 runs per model). Conclusions include:

Kimi K2.7 performed the best in tool calls, with normal parallel calls, high schema compliance, and almost no need for extra defensive parsing.
GLM 5.2 showed stable performance in long-context scenarios stuffed with 60–80k tokens.
Qwen 3.7 occasionally returned tool args as string JSON, which could be mitigated by retries.
For deeply nested schemas with many optional fields, all OSS models underperformed compared to 4o; flattening the schema is recommended.
Latency fluctuations across different providers for the same model can be larger than differences between models. Evaluations should test specific endpoints rather than just relying on leaderboards.
DeepSeek V4-Pro had average tool-calling performance in their setup and was not selected.

The final setup is: Kimi for tool steps, GLM for synthesis/generation, and GPT-4o only for the final user-facing answers; this reduced the cost per task from roughly $1.10 to $0.35.

Original post →

Agent Test: Switching Models Cuts 70% of Costs

More from coding & agent