19 latency patterns to cut non-model latency in AI applications
bibryam · x · 2026-09-05
Bilgin Ibryam maps out 19 distributed-systems latency patterns for AI applications, arguing that the model call is only one segment of the critical path — browser work, auth, gateways, app logic, DB queries and rendering all add to time-to-first-useful-response.
The patterns fall into four categories:
- Locality: bring requests and data closer
- Work reduction: eliminate repeated work
- Concurrency: run independent work in parallel
- Anticipation: do predictable work ahead of time
Each category is explained via traditional examples, then applied to AI stacks with RAG, model gateways, MCP/tool servers and orchestration loops. The core advice: measure the full path as a distribution, find the dominant percentile driver, then optimize.
More from coding & agent
- Building AI Agents: Memory, Tool Limits, and Orchestration Frameworks — mdancho84 · 2026-09-05
- A 7-step cheat sheet for building AI agents, from system prompt to evals — mdancho84 · 2026-09-05
- Claude Code v2.1.251 now saves effort levels per model via /effort or settings. — EricBuess · 2026-09-05
- Subagents Log Where Docs Fail: Docker Stuck Points and Install Gaps — lucasmeijer · 2026-09-05
- Dev Uses Subagents as Fake Users to Test His Coding Agent's Install Flow — lucasmeijer · 2026-09-05
- ChatGPT Android app hides the Codex menu but works via manually paired remote sessions — ezyang · 2026-09-05