RAG Case Study: Weak Local Models Achieve 84-98% Accuracy via Strong Retrieval

Lower-Impression-121 · reddit · 2026-08-25

The author is building a generic Document RAG system trained on complex technical rules from 80s wargames/boardgames, using BoardGameGeek forums as evaluation cases.

Key Findings:

Pipeline Efficiency: Complex questions are resolved quickly in early runs, with scores leaping from 33% to over 90% with specific adjustments.
Small Model Viability: Using weak local LLMs (8B/14B) only impacts speed due to hardware constraints, not accuracy. This proves that the ingestion and retrieval pipelines do the heavy lifting. More powerful models improve speed and conciseness but aren't required for correctness.
Domain Understanding is King: Old rules are heavily hierarchical and cross-referenced (e.g., xx.xx), making them ideal for chunking and indexing. The author used a co-occurrence graph to boost edges between nodes.
Query Handling: Users ask questions in various ways. The pipeline handles this using Hypothetical Document Embeddings to cope with grammar and bind queries to the target content.

Conclusion: High-accuracy "production-grade" RAG comes from understanding the domain (document structure and query patterns) and rigorous testing with real-world cases, rather than relying on massive LLM sizes.

Original post →

RAG Case Study: Weak Local Models Achieve 84-98% Accuracy via Strong Retrieval

More from coding & agent