RAG Support Bot Shines in Demos, Falls Apart on Real User Questions
Sad_Lavishness_53 · reddit · 2026-10-13
A developer building a RAG-based support bot for a client hit an unexpected wall: test questions look great, but real users type half sentences, typos, and questions spanning 2-3 documents. Retrieval keeps pulling wrong chunks, and the model answers confidently anyway.
What they tried:
- Smaller chunks with overlap: helped a bit, but hurt long policy docs
- Hybrid search (BM25 + embeddings): the biggest improvement so far
- A reranker on top-20 results: better quality, but adds latency
Still unsolved: multi-doc questions and getting the bot to say "I don't know" instead of guessing. The author also asks how to evaluate retrieval quality reliably without manually reading hundreds of outputs, drawing community discussion on evaluation and retrieval tuning.
More from coding & agent
- SaaS Vendors and Agencies Are Asking to Plug Into HQ, the New Entry Point for Work — jacob_posel · 2026-10-13
- He Built a 7-Bot Product Swarm on Grok Bot — Here's What Actually Works — PawelHuryn · 2026-10-13
- Claude tried installing ffmpeg from a sketchy fork instead of apt while setting up a VM — haydendevs · 2026-10-13
- Google rolling out custom MCP support in Gemini desktop app to more users — testingcatalog · 2026-10-13
- Matt Pocock launches AI Coding Dictionary defining harness, cache tokens, effort in plain English — mattpocockuk · 2026-10-13
- Bitdefender launches free macOS AI Guardian with MCP server verification for Claude Code — Bitdefender_ · 2026-10-13