Building distributed training framework from scratch
hkproj · x · 2026-08-18
极长视频教程，从第一性原理出发使用 PyTorch 构建分布式训练框架。涵盖数学原理推导及 MLA、RoPE、YaRN 等组件。讲解进程组、集合通信、分布式 Autograd、设备网格，并实现流水线并行、数据并行、FSDP、HSDP、张量并行和上下文并行等现代 LLM 训练并行技术。
More from Infra
- HBM Inventor Criticizes Samsung's zHBM: Stacking Memory on GPU Causes Melting — zephyr_z9 · 2026-08-18
- Five DGX Sparks Run Hot Serving vLLM — And It's the CPU, Not the GPU — EAccelerate_42 · 2026-08-18
- Ultra-fast Fourier transform and optical AI with a single lens — MeasurementDull7350 · 2026-08-18
- Dashboard Tracks OpenRouter Inference Provider Market Share and Revenue — Ronangmi · 2026-08-18
- Rebellions Exec on Sovereign AI: Infrastructure, Data, and Expertise — DavidBennett__ · 2026-08-18
- Harvard study: LLM serving efficiency relies on traffic patterns — rohanpaul_ai · 2026-08-18