Same model, different harness cuts cost by a third: Redditor builds live LLM cost router

leebase65 · reddit · 2026-09-13

A developer details a fully data-driven approach to controlling AI coding costs:

Metrics: Argues cost-per-token is the wrong metric; measures cost per successful task instead, using a bespoke benchmark built from his own coding data, tested by role (planner, supervisor, coder, code review) across models and harnesses.
Key finding: Running the same model in Pi vs. Codex cuts cost and time by more than a third. He continuously logs model, harness, elapsed time, and success/failure for every run.
Live routing: He built a real-time router optimizing expected cost per accepted outcome, factoring in subscription quota (much cheaper than API pricing), avoiding plans with ≤10% remaining, and falling back to pay-as-you-go APIs. Strategy: start with ultra-cheap models like DeepSeek V4.1 Flash and GLM 5.3 Flash, escalate on failure.
Subscription changes: Moved from $100/mo each on OpenAI/Anthropic/Gemini to $200 OpenAI (4x quota) plus $20 Anthropic and $20 Gemini, more than doubling usage.
The system is integrated into his AI Employee Factory platform and he plans to open-source it.

More from coding & agent