Slashing RAG Latency From 90s to 4s
ezzeddinabdallah · reddit · 2026-07-18
The author shares a RAG pipeline optimization case. Originally, a single query took 90 seconds, causing users to leave before getting an answer.
The root cause wasn't the model, but the retrieval layer itself:
- Overly heavy embeddings
- Lack of caching
- Compounding repeated calls as the document set grew
After refactoring this layer:
- Response time dropped from 90 seconds to about 4 seconds
- Costs decreased by roughly 95%
- Weaviate was used to rebuild retrieval, specifically to fix accuracy issues with "incorrect retrieved content"
The author concludes that in many AI performance issues, the real bottleneck isn't the model, but the overlooked infrastructure layer.
More from Infra
- CoreWeave 5-year CDS spikes to ~1,100bps as market fears AI bubble starts as credit event — kimmonismus · 2026-10-26
- NVIDIA Plans ~$2,000 Per GPU Per Year Software Fee for Running AI Agents — LinusEkenstam · 2026-10-26
- 12 DevOps concepts to learn in order, from Linux to SRE — _jaydeepkarale · 2026-10-26
- Killed our Vector DB for SQLite BM25 and wrote a 3.6ns Rust sentinel: lessons from multi-agent loops — OpenComplaint524 · 2026-10-26
- Gemma 4 26B-A4B runs offline on a 16 GB Mac: 33/40 on TypeScript tasks at 31 tok/s — Mission-Sir-1614 · 2026-10-26
- How thousands of GPUs train one AI: a breakdown of 5D parallelism — jbhuang0604 · 2026-10-26