30 LLM Inference & Serving Interview Questions Compiled
techNmak · x · 2026-08-21
Compiled 30 interview questions covering the systems side of running language models in production, aiming to deeply understand LLM inference mechanisms.
Key Sections:
- Inference Fundamentals: Prefill vs Decode, breaking down compute/memory boundaries, latency vs. throughput metrics.
- KV Cache & Scheduling: Memory calculation, MHA/MQA/GQA differences, Continuous Batching, PagedAttention, KV Offload strategies.
- Optimization: Quantization trade-offs, Speculative Decoding acceptance, FlashAttention, CUDA Graphs.
- Distributed & Production: Tensor/Pipeline parallelism, disaggregated prefill/decode, capacity planning, long-context inference, MoE serving.
Includes interviewer notes with follow-up probes, emphasizing that performance depends on the specific resource bottleneck.
More from Infra
- Miles v0.1 integrates Mooncake backend, accelerating remote data fetch by 14x — BanghuaZ · 2026-08-21
- AlloyDB scales vector search to 10 billion vectors with four-level tree — rseroter · 2026-08-21
- Rackspace and AMD Add 30MW of Private Cloud AI Compute for Regulated Industries — DavidLinthicum · 2026-08-21
- Developer predicts Modal will be acquired by a big lab, calling it the Stripe of agentic scientific computing — iskander · 2026-08-21
- Meta spends hundreds of millions annually on AI via Microsoft Azure — dinabass · 2026-08-21
- LFM2.5-DSpark Delivers Up to 3.2x Faster Inference — Hugging Face Blog · 2026-08-21