Raindrop launches Simulations: auto-test AI agents on every PR with real production scenarios
soleio · x · 2026-11-06
Raindrop released Simulations, a CI-like testing layer for AI agents: every PR automatically runs against real production scenarios, with the service handling sandboxing, fixtures, real/synthetic data, and eval setup.
Key details:
- 20 scenarios per run at $0.04-0.05 per run, taking 3-4 minutes
- Detects regressions in agent behavior, latency, and cost, reporting results directly in GitHub via a bot
- In one example, it caught that a lookuporder change made a support agent update the newest order when customers asked about older ones, degrading 2 of 7 affected conversations
- Caddy founding engineer Chris Catanese says it lets teams verify whether a prompt change actually fixes the intended problem before shipping
More from coding & agent
- Baker Lab + NVIDIA unveil RFD4-Proteina: agents run protein design tool calls on GFP — AllThingsApx · 2026-11-06
- New Compound Writing skill PR lands on GitHub, pending testing before merge — kieranklaassen · 2026-11-06
- Roblox brings Open Cloud to Studio MCP, letting agents manage games and assets — rms80 · 2026-11-06
- 720K public decision-training samples released for building tiny in-house decider models — MaziyarPanahi · 2026-11-06
- Astrocade Creator Competition: one winner takes the full $100K in three weeks — andrey_kurenkov · 2026-11-06
- Tackling the agent code review bottleneck: 3 steps for your agent workflows — HowDevelop · 2026-11-06