Stripe Finds Fine-Tuned 4B Open-Weight Model Beats Prompting by 24% AP on Fraud Review
emilygsands · x · 2026-10-15
Emily Sands at Stripe shares a real-world test on merchant fraud review, answering whether fine-tuning LLMs is worth it: generally yes when you need precise evaluations plus the flexibility to use rich, heterogeneous evidence.
They fine-tuned an open-weight 4B model on review labels and read a fraud score from its yes/no token probabilities. Average precision rose 24% over prompting, and the model beat human reviewers at the same recall.
Linear probes suggest a key driver is that fine-tuning made signals from legitimate businesses easier to decode. Full post linked in the original.
Related event: Stripe's fine-tuned 4B open model beats prompting by 24% in fraud review(2 posts)→
More from Research
- Microsoft's AutoEnvScaling lets agents build their own RL environments, lifting Qwen3 Terminal-Bench from 41.1 to 53.3 — omarsar0 · 2026-10-15
- Microsoft open-sources ML tool that corrects weather model bias, extending reliable forecasts to 2-6 weeks — tomwarren · 2026-10-15
- HNC paper: shifting ViT weights 1.3% keeps predictions but scrambles representations — fatihdin4en · 2026-10-15
- LLM Tool-Call Judges Reject Correct Values Without References: The Derivation Gap — Linhai Ma · 2026-10-15
- Work photos are AI's first mile: if capture is wrong, no model can recover — arnabdotorg · 2026-10-15
- Decoding NCM's latents reveals coarse-to-fine text generation trajectory — Azaliamirh · 2026-10-15