Every Citation Right, Answer Still Wrong 9/9: Code-Checked Conditions Fixed RAG to 5/6
International_Elk_33 · reddit · 2026-10-18
A developer recounts how their RAG engine answered a Canadian tax question wrong 9 times while every sentence was correctly cited and passed fact-checking — the model applied the wrong case of the rule (deducting moving costs from a scholarship instead of job pay).
The fix (0/9 → 5/6): stop letting the model judge whether a rule applies. The model rewrites each rule's conditions as yes/no questions quoting the rule's own words; answers may only copy the asker's own words, otherwise "unsettled"; code compares both quotes and decides applies / doesn't apply / can't tell.
Supporting techniques:
- A planner reads the question before searching; situation-describing questions stay whole.
- Numbers come from code: the model writes formulas over source figures; unsupported numbers are refused.
- Every sentence is verified (code word-match, separate model check, or computation); failing sentences are removed.
Cost: 15–25s and under a cent extra, only on personal-situation questions. The underlying method is documented on SSRN with a benchmark over all 34,735 FDA drug labels.
More from coding & agent
- Indie dev's Qiaomu Obsidian plugin suite tops 20,000 downloads, Agent plugin surging — vista8 · 2026-10-18
- Zeno's paradox, agent edition: Sol does half the task, then writes 'gates' reports — max_paperclips · 2026-10-18
- Fully autonomous agent with no directives goes open source, shows escape behavior — Strict-Print-7021 · 2026-10-18
- Dev runs unattended agents on Cloudflare's Pi Durable harness: scrape, write, notify, no human — MicahBerkley · 2026-10-18
- AI agents email each other and a CPA to file taxes in 20 minutes: 'we're still so early' — JasonBotterill · 2026-10-18
- Developer builds change alerts for LLM prices and retirements across Bedrock, Azure and Vertex — jsandhu0780 · 2026-10-18