Multi-agent RAG panel failed in 3 unexpected ways: irrelevant citations, self-citation loops, fixed top-K
FreakFrakFrok · reddit · 2026-10-22
The author ran three specialized agents (coding, math, RAG) plus a moderator debating one problem with citations — and it failed, covering only 1.5 of 4 success criteria. Three unexpected failure modes:
- Citations were real but irrelevant: a symbolic-integration paper justified a hashing design choice. Checking that a citation supports a claim isn't enough; relevance needs its own check.
- The system cited its own output: agent-generated notes sat in the knowledge base and got retrieved as evidence, degrading quality in a loop. Mark all generated content as hypothesis, never evidence.
- Fixed top-K retrieval: always 4 chunks meant agents cited filler when nothing was relevant. Use a relevance floor.
Bonus: the moderator invented an idea nobody proposed because the "don't invent ideas" rule only constrained structured output, not free text — enforce critical rules deterministically, not just with an LLM judge. Bigger lesson: measuring process (citation rate) instead of the task taught the system to cite anything; the real metric is whether final answers cover success criteria with relevant sources.
More from coding & agent
- Pi Coding Agent Gets a Full Rust Port via Claude, Under 90MB Memory — jasonkneen · 2026-10-22
- Securing coding agents with Docker Sandboxes and edge LLM inference at ATO — HowDevelop · 2026-10-22
- Free book 'LangGraph in Small Pieces' teaches agents from blocks to production — truckkicker · 2026-10-22
- What Everyone Is Getting Wrong About Context, per a Founder's New Article — shensi · 2026-10-22
- Cloudflare's AI security-audit skill tops GitHub with 38.5k stars — alex_verem · 2026-10-22
- PMs are putting CLAUDE.md, skills and evals in GitHub: 3 repos, 4 workflows only git enables — aakashgupta · 2026-10-22