How a 50-customer B2B app automated QA with agents simulated from production MLflow traces
Lopsided_Feedback_81 · reddit · 2026-11-20
A dev running a B2B data-analysis app (50 customers, 2-10 users each) shares how they automated QA with AI agents:
- Production agent traces (prompts, tool calls) were already logged to a self-hosted MLflow instance on Cloudflare.
- They sampled sessions and extracted real user queries into a 'question bank' as template generator.
- Multiple agents each simulate a real store's user, asking randomized bank questions applied to their own store data; the app runs normally while tracing tool calls, timings and responses.
- A separate agent then evaluates each conversation trace for factual accuracy, speed, and cost.
- They added a zone between staging and production — a private beta where every change must pass this QA gate, ensuring pushes at least don't degrade current performance.
They admit the simulation isn't perfect and unseen cases can still break, but models are now smart and cheap enough that automated agents beat their previous time-consuming manual QA.
More from coding & agent
- 200 lines of Python + Gemini generates a synced-caption, voice-cloned walkthrough video — DynamicWebPaige · 2026-11-20
- Strategist evolves reasoning strategies as code, wins 24/30 benchmarks at 39% lower cost — liuzhuang1234 · 2026-11-20
- RestTemplate is deprecated in Spring Boot 4.2 — here's the direct swap to RestClient — therealdanvega · 2026-11-20
- Developer gives Claude Code its own computer to drive browsers, spreadsheets and desktop apps — jradoff · 2026-11-20
- Garry Tan shares GBrain cases: AI working overnight, 850 pages in 3 weeks — garrytan · 2026-11-20
- Perplexity agent spent $2.20 screening 327 land parcels and produced a 4K site-report video — garrytan · 2026-11-20