Stripe Scales Fraud Model Shield with Agent-Assisted ML, Recall Up Over 8%

Stripe's engineering team detailed the full scaling journey of its transaction fraud model Shield in a blog post, "Scaling Radar: Fighting fraud with agent-assisted machine learning": after a fraud attack bypassed Shield, the team let agents take over the ML experimentation loop. The winning setup turned out to be a better stochastic weight averaging (SWA) configuration, which lifted model recall by more than 8% after scaling and now blocks tens of millions of dollars in fraud losses for merchants each year. The case demonstrates a viable path for agents and engineers to collaborate on hunting fraud patterns, with lessons applicable to other ML systems.

Confirmed

• The trigger was a fraud attack that bypassed Shield; agents used model explanations to generate hypotheses, the team filtered for the most promising ones, and agents then validated them with tests
• Post-scaling recall improved by over 8%, saving merchants tens of millions of dollars in fraud losses annually
• Single-epoch training time dropped from 24 hours to 4.8 hours with no dataset reduction, using techniques like S3 streaming reads and torch.compile
• The team wrote operational knowledge into documents for agents to load, enabling them to run the full experimentation loop: explore data, build features, backfill history, train models, debug failed jobs, and evaluate against baselines
• The team's conclusion: with agents running the experimentation loop, engineers can focus on judgment-heavy work — setting priorities, reviewing evidence, and deciding what ships
• Radar's fraud detection relies on network-wide signal correlation — e.g., a new card seen at one merchant may be linked to risk records at other Stripe merchants — and more complete historical data is key to improving recall

Why it matters

• This is a public case of agents end-to-end taking over an ML experimentation loop with quantifiable returns (recall +8%, tens of millions in fraud blocked annually), validating the engineering value of human-machine division of labor
• Three transferable lessons: iteration speed determines learning speed (5x faster training), documenting operational knowledge so agents can act on it, and boosting recall with fuller historical data and network-wide signals

2026-10-20 ~ 2026-10-20 · 7 related posts

• Stripe scaled its fraud model Shield and improved recall by over 8% — emilygsands · 2026-10-20
• [source] Stripe cut fraud model training from 24h to 4.8h per epoch without shrinking data — emilygsands · 2026-10-20
• Stripe's recipe for agents running the full ML experiment loop — emilygsands · 2026-10-20
• Stripe: agents free engineers for judgment work, not more toil — emilygsands · 2026-10-20
• How Stripe responded when fraud got past Shield: agents generate, humans pick — emilygsands · 2026-10-20
• [source] Stripe: agents now run the ML experiment loop, blocking tens of millions in fraud a year — emilygsands · 2026-10-20
• [source] Stripe details scaling Radar: fuller training history plus agent-assisted ML cut fraud — emilygsands · 2026-10-20