Real-World Test: Switching Models for Production QA Agents

dumbfoundded · reddit · 2026-07-14

Running browser-driven QA agents in production, the author migrated their highest-traffic agent from GPT-5.3-codex to MiniMax M3. Using step-by-step billing data, they conducted a natural experiment spanning over 5e11 tokens.

Key findings:

Median cost per pipeline dropped by roughly 55%;
MiniMax M3 requires about 2.5x more tokens for the same work, but the final bill is cheaper due to lower unit pricing;
The same model performs vastly differently across inference providers, with prompt cache hit rates and reliability significantly impacting actual costs.

By reverse-engineering the bills for an "effective unit price," the author discovered that codex's output tokens incur actual costs noticeably higher than the sticker price, caused by a mismatch between reasoning token billing and visible token statistics. The author emphasizes that evaluating agents requires looking beyond token efficiency to consider real costs, failure rates, retry overheads, and provider implementation details.

Original post →

Real-World Test: Switching Models for Production QA Agents

More from coding & agent