Comet Builds Docker-Based Incident Testbed to Evaluate Remediation Agents with Opik
craigsdennis · x · 2026-11-11
Comet published what fullstackpython calls one of the best recent engineering posts on agent evals: a demo environment inspired by financial services used to stress-test an incident remediation agent.
- Testbed: a typical payments stack (API gateway, payments service, fraud scoring, card processor routing to two acquirers, Postgres-backed ledger, FX rates service, notifications, Redis idempotency cache) with Prometheus, Loki, and Grafana supplying metrics, logs, and alerts, all Docker-based.
- Method: the team deliberately broke the stack in ten ways (4 variants each) to trace how agents investigate and respond to incidents.
- Tooling: their open-source platform Opik handles tracing and evaluation; the Opik Agent Optimizer improves performance in a few clicks.
- The post stresses that agent mistakes in financial infrastructure (wrong rollbacks, paused merchant payouts, disabled fraud checks) are costly, making visibility and continuous evaluation essential.
More from coding & agent
- LangChain Launches Annual State of Agent Engineering Survey — hwchase17 · 2026-11-11
- RX 6800 gets 3x faster ComfyUI images by re-enabling silently disabled INT8 — TheCat001 · 2026-11-11
- AutoRubric 1.7.0 grades images against rubrics, prompt caching cuts per-criterion cost ~25% — deliprao · 2026-11-11
- Replicate a 4M-view fitness video formula with Claude skills, step by step — aigleeson · 2026-11-11
- Cursor said to be building multiplayer mode letting teams and their agents share a workspace — testingcatalog · 2026-11-11
- Google wires CodeMender into pull requests: /fix comments auto-patch vulnerabilities — rseroter · 2026-11-11