Weight Teleportation makes cross-cluster RL weight updates faster than intra-cluster ones
Kyrannio · x · 2026-11-03
- Problem: In large-scale RL, shipping weight updates from trainers to samplers is slow and expensive, so datacenters and RL algorithms assume both must be co-located — the core drawback of decentralized RL.
- Approach: The humansand team introduces weight teleportation, using GPUs for massively parallel compression of weight deltas plus a heterogeneous decompression and patching pipeline co-designed with hardware-native features.
- Results: Sampler pause time under 0.23s, 3s of non-blocking compression + prep; cross-cluster updates are often faster than intra-cluster ones, with the bottleneck now the GPU's decompression ASIC. Samplers can live anywhere — "maybe even space."
More from Infra
- Compute growth driven entirely by R&D spend; customer-serving compute stays flat — teortaxesTex · 2026-11-03
- RTX 3090 tensor unlock on Windows finally boots: BitLocker fix, BF16 70→115 TFLOPS, 20% faster video gen — LavateLasManos666 · 2026-11-03
- Databricks: Agents now run AI research loops, training jobs grew 8x in a year — matei_zaharia · 2026-11-03
- RTX 5080 reportedly discontinued, store prices nearing $2,000 — PAstynome · 2026-11-03
- Inference startups hitting $100M+ ARR are just reselling prebooked compute, not real businesses — abhiadesai · 2026-11-03
- Stripe Data Pipeline adds real-time sync to Lakebase Postgres with seconds-level updates — matei_zaharia · 2026-11-03