Treating LLMs as unreliable dependencies: system design for production AI
_jaydeepkarale · x · 2026-09-24
A reposted recommendation highlights a book connecting classic system design to LLM applications: most LLM material stops at prompts and model choice, which gets you through a POC but not production, where latency, cost, failures, security and unpredictable outputs dominate. The book frames LLMs as an expensive, slow, non-deterministic dependency and prescribes real engineering discipline — gateways, circuit breakers, caching, model routing, evaluation, retrieval pipelines, observability and human review. Case studies include an adaptive learning platform that generates content offline, routes it through human review, and pre-builds personalized lessons. Useful for anyone shipping LLM systems to production.
More from coding & agent
- Splitting sandbox base layers with Nix: fewer images, more auditable agent environments — sloppenheimer · 2026-09-24
- Devin adds native Teams support and first-party Microsoft 365 integration — DevinAI · 2026-09-24
- Dev on AI sandbox tooling: audit chronus at syscall/eBPF layer, hide extraneous tools — sloppenheimer · 2026-09-24
- Notion Engineer: Subsidized Tokens Mean You Should Switch Agents Freely Without Losing Context — nbaschez · 2026-09-24
- Unverified claim: 'GPT-6 Astra' builds full video projects via Codex + Dreamina CLI — JaynitMakwana · 2026-09-24
- Open-Source Rust/CUDA Engine reflex Adds DeepSeek MLA, Cold-Start 1.3x Faster Than llama.cpp — SaveAmerica2024 · 2026-09-24