Many 'dumb agent' complaints are retrieval problems: RAG tool scores
rcleffke · reddit · 2026-10-30
Scoring AI agent tools for a side project, the author finds many 'the model is dumb' complaints are actually retrieval failures: wrong document, stale document, or nothing.
Their current tool scores (out of 10, weighted by builder fit, maturity, release freshness, value):
- LlamaIndex: 7.0 — strongest for RAG pipelines and document agents
- Pinecone: 5.9 — solid builder/maturity scores, held back by no recent tracked release
- Ragas (eval, not storage): 5.2 — free, but last tracked release was in January
The author shares a free rankings site and asks readers how they evaluate retrieval separately from final answers in production.
More from coding & agent
- Anthropic launches Claude Duet, a proactive agent powered by Opus 5.5 — joellewenstein · 2026-10-30
- Spring AI 2.1 adds VectorStore.upsert for apps that bring their own embeddings — therealdanvega · 2026-10-30
- Auditing an LLM-built knowledge graph: 0 of 186 facts matched verbatim, 1/3 of edges hit wrong entities — JeremyCMorgan · 2026-10-30
- Same agent harness, opposite rankings: TB4 benchmark drama exposes eval chaos — lunwang1996 · 2026-10-30
- The eternal eval debugging meme: ask the vendor if it's data or harness, it's data — dejavucoder · 2026-10-30
- DropCut: Open-Source ComfyUI Rebuild of Higgsfield Turns Photos + Song Into Beat-Synced Reels — Mediocre-Purchase-62 · 2026-10-30