Support bot promised free repairs: a RAG hallucination postmortem
CommonMinimum587 · reddit · 2026-10-18
A company's self-hosted RAG support bot blended an extended-plan doc with a customer's actual warranty and promised free repairs they had to honor (citing Air Canada's 2024 tribunal loss). Logs looked fine—the answer was fluent, just built on the wrong chunk. Their mitigation split: log retrieved chunks per answer (Langfuse, Arize Phoenix) for tracing, plus pre-send guardrails blocking coverage/refund claims not backed by evidence (Orq AI on-prem evaluators vs. Nemo Guardrails). Open question: whether outbound guardrails beat daily transcript reviews.
More from coding & agent
- Thorsten Ball: A 'wrong' code push matters far less now that AI makes changes cheap — sloppenheimer · 2026-10-18
- Every trains a Slack agent to edit like its editor-in-chief using a custom skill — every · 2026-10-18
- jedisct1: cross-language calls need just a serialization format and plain FFI, not WASM — jedisct1 · 2026-10-18
- 7 practical rules to drastically improve Claude's writing, from iterative drafting to cross-model review — EXM7777 · 2026-10-18
- 'Agent Psychiatry': evaluating skill effectiveness via thinking traces — heyneighbor · 2026-10-18
- OpenCode discloses breach from large-scale distillation attack hitting 1.4% of users — TheZachMueller · 2026-10-18