Meta open-sources spmd_types: a type system to verify distributed PyTorch training correctness
austinvhuang · x · 2026-09-19
meta-pytorch/spmdtypes, inspired by JAX's sharding-in-types and adapted for PyTorch, offers two type systems: Local SPMD types for safely using Megatron-style differentiable collectives by tracking pending gradient reductions, and Global SPMD types, a DTensor-like abstraction with explicit communication ops so code behaves identically on one device or many. It lets you statically check gradient correctness or parallelization equivalence without running a full E2E distributed training job.
More from Infra
- Open models now outspend OpenAI on Vercel gateway, taking 78% of token volume — soumitrashukla9 · 2026-09-19
- Multi-GPU rig cooling debate: should fan curves be tuned per GPU under load — TheZachMueller · 2026-09-19
- ROCmFix: zero-dependency Python tool auto-fixes AMD ROCm setups and benches Vulkan vs HIP — pavlexlkl · 2026-09-19
- Emulating memory access: FEX-Emu devs on the x86-to-ARM memory model minefield — blaizedsouza · 2026-09-19
- Three lines of bash escape Docker's Mac hypervisor: CVE-2026-77179 patched in Docker Desktop 4.88.0 — blaizedsouza · 2026-09-19
- Cosmos DB VLDB paper shows better scheduling can be worth $100M+ a year — blaizedsouza · 2026-09-19