Small models ace payout fraud review in Kill Bill — same answers, 106x cost gap vs Claude
m4rkGS · reddit · 2026-11-05
A developer tested small models on real finance ops: reviewing 10 pending payouts in open-source billing system Kill Bill locally, deciding release/hold/block for each. Total runtime 4.6s: 4 released, 3 held ($5.2k) pending card settlement, 3 blocked ($11k) — including a chargeback and two account takeovers — with an explanation note left on each account. 10/10 matched the answer key.
Running each case 3x across three models, accuracy was identical (30/30) but costs diverged sharply: drex $0.00018 per 10 payouts at 0.9s, gpt-6 luna $0.0008 at 1.5s, and claude sonnet 5.5 $0.019 at 3.6s — roughly 106x more expensive for the same answers, and slower.
The author's takeaway: these tasks hinge on structured signals (chargeback history, recent account changes, card settlement), so frontier models aren't needed; they'd likely pull ahead only on messier cases with contradictory signals. Previous test: loan decisions; next up: bank reconciliation in ERPNext.
More from coding & agent
- Consultant replaces $1,300/mo 'AI employee' agencies with a Claude playbook that recovered $167K — aryanXmahajan · 2026-11-05
- Open-source product-engineering skills ship real products with coding agents, no coding — PawelHuryn · 2026-11-05
- Gemini CLI v0.67.0 ships IDE companion container sandbox bridge and a batch of fixes — gemini-cli-robot · 2026-11-05
- Cloudflare maps out x402 payments so AI agents can pay for the content they read — kleffew94 · 2026-11-05
- Running Qwen at 1Q quant: dev shares DeepSeek Harness fix plus Nemotron 3.5 tips — gnukeith · 2026-11-05
- Perplexity launches MotionSites skill so Computer can build WebGL-animated websites for all users — AravSrinivas · 2026-11-05