A six-case smoke test to separate missing vs conflicting evidence in document Q&A
Heavy-Confidence1826 · reddit · 2026-11-23
The author proposes a six-case starter fixture for document-Q&A answer steps, designed to expose two failures that averages hide: answering when sources are silent, and picking a side when sources conflict.
- Delivery-FAQ-only context should yield "insufficient evidence," not a guessed return window.
- Two conflicting policies (30 vs 21 days, no replacement clause) should be reported as a conflict regardless of context order.
- With an explicit replacement clause, an April purchase gets 30 days but a February purchase still gets 14 — don't blindly prefer the newest record.
Score answer, citation support, and uncertainty handling separately per row; keep row results visible instead of burying conflict failures in an average, and record model, prompt, and context order while repeating packets to check variability.
The fixture is explicitly fictional and a smoke test only — it doesn't measure retrieval quality, access control, or full-system reliability. The design applies OpenAI's evaluation best-practices guide.
More from coding & agent
- Apple cuts per-dev daily token budget from $300 to $100; cheaper-model strategies follow — Zachly · 2026-11-23
- Fine-tuned Gemma 4 E4B drives 100 villagers offline in a rumor-spread simulation game — AlternativeEnough352 · 2026-11-23
- Nous ships official Hermes Blender plugin: agents drive live 3D scenes via MCP — Teknium · 2026-11-23
- Dev builds Battlefield 1942-style browser FPS with GPT-6.2: 16v16 battles, vehicles, AI squads — VraserX · 2026-11-23
- Dev spins up hundreds of AI agents in a livestreamed attempt to solve P = NP — amantinband · 2026-11-23
- A 9-machine local AI homelab: 5 agents on one dual-3090 llama-server — StandardLovers · 2026-11-23