4 essential reads to understand LLM decode from first principles
CatAstro_Piyush · x · 2026-11-21
A curated list of 4 classic reads for understanding LLM decode from first principles: kipply's Transformer Inference Arithmetic (KV cache, capacity, tensor parallelism, batch sizes, FLOPS counting, benchmark comparisons), Flash-Decoding by Tri Dao et al., PyTorch's gpt-fast writeup, and the original speculative decoding paper. Ideal for engineers who want to reason about inference latency, memory bandwidth, and parallelism costs arithmetically.
More from Infra
- Microsoft launches MAI Licensed Weights; Purdue researcher asks to run them on own hardware — marlene_zw · 2026-11-21
- Looped CED cuts KV-cache memory 36.8x on 552B model, but encoder loops cost compute — teortaxesTex · 2026-11-21
- Agents are sessions, not requests: Google explains sizing GKE node pools for agent workloads — rseroter · 2026-11-21
- AI and chip stocks plunge: Micron -12.6%, Arm, CoreWeave, Anthropic all down over 10% — xiaohu · 2026-11-21
- Ben Bajarin: AI is the deployment phase of the Information Age, investors unconvinced — BenBajarin · 2026-11-21
- High-Flyer did try to obtain 75K GPUs legally, per teortaxesTex — teortaxesTex · 2026-11-21