System Design for AI Day 2: back-of-envelope math sizes an LLM app at $18K/month

techyoutbe · x · 2026-10-20

techyoutbe's "System Design for AI" series Day 2 covers requirements and capacity estimation with a detailed infographic.

Functional vs non-functional requirements: what the system does (QA, summarization, generation) vs how well (p95 latency, QPS, accuracy, uptime, cost)
Clarify before designing: users, expected QPS, acceptable latency, accuracy bar, budget
Back-of-envelope example: 1M daily requests ≈ 12 avg QPS, 60 peak QPS; at 1K tokens/request that's 30B tokens/month, roughly $18,000/month at $0.60/1K tokens
Capacity diagram: QPS × tokens → GPUs/VRAM; logs and data → storage
SLOs/SLAs: set realistic targets first (p95 ≤2s, ≥99.9% uptime), then design to meet them
Cost framing: $/1K tokens, $/GPU-hour, cache hit savings (30% hit rate ≈ 30% less cost)
Pitfall: designing before sizing leads to over- or under-provisioning; includes a practice task for a 500K daily query app

More from coding & agent