How Comet improved a financial incident remediation agent's score from 0.34 to 0.57 with Opik
dl_weekly · x · 2026-11-17
Comet shares a hands-on guide to evaluating and optimizing an AI incident remediation agent for financial services using Opik. The demo spins up a realistic payments stack (API gateway, payments service, fraud scoring, card processor, Postgres ledger, FX service, Redis idempotency cache), deliberately breaks it in ten ways, then traces and scores how the agent investigates and resolves each incident. Using the Opik Agent Optimizer, the team raised the agent's overall score from 0.34 to 0.57 across 30 validation runs while eliminating unsafe actions like wrong-service rollbacks or disabled fraud checks.
More from coding & agent
- Grok Bot testing live phone call handling: type replies in chat, bot relays to caller — XFreeze · 2026-11-17
- Foaster partners with consultancy Kéa to bring agentic AI into consulting — RaphaelDabadie · 2026-11-17
- Stitch embeds AI agents in banking: the safety story is harnesses, not hack-proofing — alex_verem · 2026-11-17
- Agent Identity 101: one sequence diagram explains delegated, agent, and MCP auth patterns — adnan_hashmi · 2026-11-17
- Running Voice Agents in Production: Dev Shares 1,200 Calls/Day and 5,000 Calls/Month Setups — FullStackFrenzy · 2026-11-17
- Meta's Self-Improving Memory: agents discover their own memory skills and distill them into weights — jaseweston · 2026-11-17