AI Demos Flop in Production Because You Picked the Examples: Build a 50-100 Case Eval Set First
salespire · reddit · 2026-10-30
A developer breaks down why AI features demo well but flop in production: demos rely on 15 hand-picked clean inputs, while real users send scanned 40-page PDFs, compound questions, typos, half sentences and other languages — you're just seeing the real input distribution for the first time.
The boring fix that works: collect 50-100 real inputs (especially ugly ones) before writing the first prompt, write down what a good answer looks like for each, and score every change against that set.
Side benefits:
- You find the 10% of inputs causing 90% of the pain and can route them to humans
- You can answer "is this version better?" with data instead of gut feeling
Most teams skip this homework, then do it anyway three weeks later under pressure.
More from coding & agent
- Stacked logical commits: how one team structures AI-generated optimization PRs for cherry-picking — jmschreiber91 · 2026-10-30
- AI-Generated Mega-PRs vs Small Commits: Hours-Long Test Suites Block Review — jmschreiber91 · 2026-10-30
- Author: Mega AI PRs Aim to Prove Perf Wins, Not Get Merged As-Is — jmschreiber91 · 2026-10-30
- TextFold: open-source reader keeps LLM summaries one click from source text — Lopsided_Scarcity979 · 2026-10-30
- Autonomous agents negotiating, coding contracts, trading tokens end-to-end — curious_vii · 2026-10-30
- Boson Studio open-sourced: conversational LLM-driven image and video generation — clarvalon · 2026-10-30