A six-case smoke test to separate missing vs conflicting evidence in document Q&A

Heavy-Confidence1826 · reddit · 2026-11-23

The author proposes a six-case starter fixture for document-Q&A answer steps, designed to expose two failures that averages hide: answering when sources are silent, and picking a side when sources conflict.

Score answer, citation support, and uncertainty handling separately per row; keep row results visible instead of burying conflict failures in an average, and record model, prompt, and context order while repeating packets to check variability.

The fixture is explicitly fictional and a smoke test only — it doesn't measure retrieval quality, access control, or full-system reliability. The design applies OpenAI's evaluation best-practices guide.

Original post →

More from coding & agent

coding & agent channel →