Engineering postmortem: False 100% success rates and hidden latency disasters
CupGlass540 · reddit · 2026-08-25
The author shares practical lessons from troubleshooting a self-built model gateway, revealing serious performance issues hidden behind perfect monitoring metrics:
- Backend Queue Explosion: After removing concurrency limits, the gateway reported 100% success (HTTP 200), but 31/32 responses arrived after the client gave up. With limits, success dropped to 25%, but valid responses increased 6x.
- Failover with Uncontrolled Latency: Failover logic only checked liveness, not latency. A slow backend (3s) was constantly called while a healthy one was ignored. The fix involved a global request deadline, limiting each backend to 60% of remaining time.
- Bottleneck Misjudgment: The suspected DB read hotspot was only 0.146ms; the real bottleneck was metric row flushing to disk (75% time). Optimization reduced latency from 2.999ms to 0.230ms.
Conclusion: Success rate is a misleading metric that masks outages. The key metric is the number of valid answers that arrive while someone is still waiting.
More from Infra
- 4070 Ti Benchmarks: Running Kimi K3, DeepSeek V4, and Qwen 122B Locally — JayB_Official · 2026-08-25
- CXMT LPDDR6 revealed, Xiaomi XRING-03 as first adopter — teortaxesTex · 2026-08-25
- Fork of h3.c adds 3090, multi-GPU, GGUF support, and a UI — QuixiAI · 2026-08-25
- 1 hour of AI work now costs less than a minute of SF parking — stuffyokodraws · 2026-08-25
- Qwen 3.8 27B on Mac hits 113 TPS via Dflash2 optimization — TheMoonMidas · 2026-08-25
- TileMix: Tile-Centric Mixed-Precision Attention for LLM Inference — Hanzhi Zhang · 2026-08-25