3,000 tables, one RAG pipeline: a payments compliance assistant hits 97% retrieval hit rate
AI Engineer · youtube · 2026-10-14
At AI Engineer, Eleonora Drykina and Nina Žižakić of In The Pocket described a production RAG assistant for an anonymous Belgian payments client, built on 270 PDFs and 6,000 pages where much of the answer-bearing content sits in tables. Key engineering points:
- Indexing: preserve section paths and chunk order so longer passages can be reconstructed; convert tables to CSV while keeping heading-value relationships
- Retrieval: dense retrieval + BM25 raised the hit rate on 63 test questions from 92% to 97% (measuring retrieval, not full answer correctness)
- Agent: four tools widen the search stepwise (results → sections → full docs → multiple rulebooks); context limits prevent oversized rulebooks from flooding the prompt, and the prompt asks the agent to reconsider further tool calls
- Latency: median answer takes 38 seconds; streaming Markdown separately from the completed source card avoids waiting on one large structured object
- Evaluation: traces, citation checks, and Promptfoo evals — including apparent regressions from the automated judge that didn't survive inspection
Early internal rollout reached 34 users and 900 questions; external release is still planned. Lessons: reconstructable evidence, bounded retrieval, visible progress, and evaluations you can challenge against the source documents.
More from coding & agent
- AWS: Bedrock AgentCore supervisor agent generates client decks in under 90 seconds — AWS ML Blog · 2026-10-14
- Tamnoon's four-agent system on Bedrock cut critical cloud exposures 90% in 90 days — AWS ML Blog · 2026-10-14
- Gemini API runs multi-hop continuations server-side with polling, step aggregation and thinking levels — clmt · 2026-10-14
- Gemini 4 Argon Built a Fully Multiplayer 3D Capture-the-Flag Game in Cursor — chongdashu · 2026-10-14
- PM Skills: open-source repo packs 69 skills and 42 commands for PMs into Claude Code — PawelHuryn · 2026-10-14
- theo ships dozens of weekly perf fixes to T3 Code, asks why AI labs don't obsess over app performance — DanielLockyer · 2026-10-14