$5 in evals cut his AI bill 74%: Claude Haiku 5.5 beat Gemini 3.8 Flash at 1/4 the price
MicahBerkley · x · 2026-10-29
A founder running a meeting-to-Asana agent (2,156 calls/month) blind-tested models on 10 real calls: Claude Haiku 5.5 caught 90% of promises vs Gemini 3.8 Flash's 57%, with 98% verbatim quoting and $0.0095/call vs $0.037. On ticket writing, graders preferred Haiku 24:6. His method: re-run evals every 1-2 months with identical inputs and instructions — $5 total spent, biggest AI bill down 74%.
More from coding & agent
- Linear's Slack bot slammed as "GPT-3.5-level" — pick your team's AI by daily reliability, not leaderboards — sujingshen · 2026-10-29
- Claude Code taps Oxylabs via MCP to scrape bot-protected sites into clean JSON — mhdfaran · 2026-10-29
- opencode ships /goal command for coding agents — lyraxana · 2026-10-29
- Make0 Brain: an open-source MCP server for shared, versioned agent memory with 80ms retrieval — heysrb · 2026-10-29
- Discover API lets AI agents search social platforms in one call, with spend caps and semantic filtering — dooddyman · 2026-10-29
- Dev's code merged by AI agent with zero human intervention — iamsahaj_xyz · 2026-10-29