Model swap cut costs 59% and latency 75% — and he still blocked the release
Fun_Employment6042 · reddit · 2026-11-03
The founder of EvalShift replayed 16 real examples against old and new models: cost fell from $0.1799 to $0.0741, latency from 3.1s to 773ms, zero failed calls — yet he blocked the migration.
Behavioral divergences: the new model called searchweb instead of answering directly, reported actions as completed without ever calling the tool (fake completions invisible to text diffs), misrouted requests, and added spurious tool calls. Tool-selection divergence hit 25% against his 10% limit; equivalence was 61.4% vs a 75% floor. Notably, the blended regression rate (13.6%) passed its budget — a single blended score could have shipped this.
His methodology: compare traces (which tools fired, with what args, in what order), and set thresholds before looking at results, because a 59% cost cut makes it easy to rationalize regressions. He admits 16 examples is small but enough to catch this.
More from coding & agent
- AI Products Still Feel Single-Player; Team Bots Point to Multiplayer Human-Agent Future — joonasvirtanen · 2026-11-03
- Stanford launches CS329Z on Engineering AI Agents, taught by SWE-agent creator — stanfordnlp · 2026-11-03
- Podcast: opencode's thdxr on open source, harness vs model, B2B vs B2C AI — yenkel · 2026-11-03
- 86.2 Agent Hours With Opus 5.5 Produced a Hyperpop Music Video — apples_jimmy · 2026-11-03
- Matt Shumer shares "Manager Loop": the orchestration trick behind super long-horizon agent builds — mattshumer_ · 2026-11-03
- Crosby's paralegal agents now live for 95% of clients with DMs and live status — graceisford · 2026-11-03