Meta open-sources spmd_types: a type system to verify distributed PyTorch training correctness
austinvhuang · x · 2026-09-19
meta-pytorch/spmdtypes, inspired by JAX's sharding-in-types and adapted for PyTorch, offers two type systems: Local SPMD types for safely using Megatron-style differentiable collectives by tracking pending gradient reductions, and Global SPMD types, a DTensor-like abstraction with explicit communication ops so code behaves identically on one device or many. It lets you statically check gradient correctness or parallelization equivalence without running a full E2E distributed training job.
More from Infra
- Malaysia poised to become a compute superpower with both US and Chinese chips by next year — basedjensen · 2026-11-04
- ByteDance details USDB storage system: 85.8% lower write latency for recommender sequences — _reachsumit · 2026-11-04
- Oracle cancels $7B chip leasing deal with Tencent, FT reports — SumitGup · 2026-11-04
- Inference engine shrunk to 1023 bytes, still runs SmolLM2-135M — cneuralnetwork · 2026-11-04
- Where Residential Proxy IPs Come From: AMA with Massive's CEO — Dylan-Hayes · 2026-11-04
- Musk says space data centers are 'looking pretty appealing right now' — elonmusk · 2026-11-04