Troubleshooting Intermittent High Latency in RAG Chatbots
arjunssat · reddit · 2026-07-10
The author is troubleshooting an intermittent 4–5 minute latency issue in a production-grade RAG chatbot.
The tech stack includes FastAPI (Python), Amazon Bedrock, and PostgreSQL + pgvector, all deployed in the same AWS region. The FastAPI container runs on Kubernetes, while PostgreSQL is on EC2. The author has already optimized database indices, pgvector indices, connection pools, thread pools, and K8s HPA, yet extremely slow requests still occur sporadically.
Symptoms:
- Local execution of the same query typically completes within 30 seconds.
- In the hosted environment, the same request occasionally drags on for 4–5 minutes.
- The issue is intermittent and independent of load; it happens even when the author is the sole user during testing.
The author plans to add instrumentation to measure latency at each stage, focusing on isolating the bottleneck among DB retrieval, vector search, Bedrock calls, and response generation. They are also seeking further troubleshooting suggestions from the community.
More from Infra
- oMLX 0.5.2 adds Mac menu-bar stats, low-bit decode kernels, and faster downloads — awnihannun · 2026-07-22
- Strangeworks launches Aura to turn enterprise ops into production optimization systems — whurley · 2026-07-22
- Graph workload 854.graph500 enters SPEC CPU 2026 as a new CPU benchmark — Prof_DavidBader · 2026-07-22
- Hybrid and local inference are emerging as a response to AI energy and token costs — dmitry140 · 2026-07-22
- NVIDIA details Vera CPU with 2x performance claims and a 22,000-core rack — ryanshrout · 2026-07-22
- NVIDIA says Vera Rubin NVL72 delivers 10x more tokens per megawatt than Blackwell — nvidia · 2026-07-22