RAG Pitfalls: LLM Setup Took 2 Hours, Vector DB Took 3 Weeks

Cautious_Bit_8521 · reddit · 2026-08-14

A backend developer shares a painful debugging journey when building a local RAG (Retrieval-Augmented Generation) setup. While configuring the LLM (Ollama + LangChain) took only 2 hours, managing the vector database for 50,000 documents consumed 3 weeks.

Database Trials & Pitfalls:

Chroma: Worked well for 100 docs, but query latency spiked to 4 seconds when scaled to 50k vectors.
pgvector: Easy to integrate, but queries were doing full sequential scans because the HNSW index wasn't properly built with efconstruction.
Milvus: Self-hosting required 14 Docker containers. Despite fast queries, it suffered from Out-of-Memory (OOM) errors during index building and complex scalar filtering syntax. The developer eventually abandoned self-hosting for managed Zilliz Cloud, resolving issues in 20 minutes.

Key Lessons: The real bottleneck of RAG isn't the AI model, but data retrieval. Chunking strategy matters more than the embedding model, and index parameter tuning matters more than chunking. Plan your vector DB setup before touching the LLM for large datasets.

Original post →

RAG Pitfalls: LLM Setup Took 2 Hours, Vector DB Took 3 Weeks

More from coding & agent