96.8% of eval labels came from the model itself — fixing bias cut costs 86%
AWS ML Blog · rss · 2026-10-28
AWS ML Blog's engineering retrospective with Property Finder: when migrating vision AI workloads to Amazon Bedrock, they found 96.8% of evaluation labels were inherited from the incumbent VLM's predictions rather than human verification — the answer key was written by the model being replaced, plus hidden label errors (hallucinated watermark text, forced single labels on multi-room images). The fix: 356 expert-annotated UAE images (256 dev / 100 eval, plus 230 Egypt cross-check), a switch to multi-label classification, and metrics led by Micro F1 with Macro F1, per-label P/R and IoU. They then ran prompt optimization as reinforcement learning — prompt versions as agents, Micro F1 as reward, error analysis as gradient — iterating v0→v27 with GEPA-like manual loops and DSPy's MIPROv2. Result: higher accuracy at 86% lower cost. The methodology and prompt-optimization loop architecture are directly reusable for your own eval workloads.
More from coding & agent
- CodeRabbit Secure adds cross-repo Deep Scan, sandbox exploit verification and merge blocking — nevrekaraishwa2 · 2026-10-28
- Hyphenate raises $45M Series A after 86x revenue growth as agents run 90% of finance work — ahuja_priyank · 2026-10-28
- Open-source anymove turns one AI video into motion capture for any creature at $0.26 per clip — noahsolomon · 2026-10-28
- When to stop correcting an AI agent and restart the task instead — ZinchaDEV · 2026-10-28
- MASS paper: one model plays the whole agent team to bootstrap recursive self-improvement — idanbeck · 2026-10-28
- Dev builds AI-native macOS dock where an agent classifies activity and notifications in real time — trq212 · 2026-10-28