How PyTorch Generates Random Numbers in Parallel on GPUs: A Deep Dive into Philox
abhi9u · x · 2026-09-25
Abhinav Upadhyay published a deep dive on how PyTorch's torch.randn() generates millions of random numbers in parallel on GPUs.
- The problem: training and inference rely heavily on randomness (weight init, dropout, sampling, SGD), but traditional PRNGs like xorshift are sequential state machines that don't parallelize across thousands of GPU cores.
- The solution: PyTorch uses Philox, a counter-based RNG where each thread derives its sequence independently from its thread ID and counter — fully parallel yet perfectly reproducible.
- The article dissects PyTorch's C++ and CUDA implementation to explain how Philox works and how parallel generation is achieved.
Related event: How PyTorch generates parallel random numbers on GPUs with Philox(2 posts)→
More from Infra
- Liquid AI ships LFM2.5-2.6B, a small fast model built for on-device agents on phones and laptops — helloiamleonie · 2026-09-25
- LFM2.5-2.6B matches Qwen3.5-9B at 3x size, unlocking on-device agents — helloiamleonie · 2026-09-25
- On-device agents need 3 things: ~4GB usable RAM, prefill speed, consistent quality — helloiamleonie · 2026-09-25
- Australia to host large share of next-gen datacenters, built for Anthropic — mattbeane · 2026-09-25
- Goldman: hyperscaler capex to grow 54% next year to $1.2 trillion — firstadopter · 2026-09-25
- Fine-tuning a 194M GLiNER2 dataset tagger on HF Jobs costs $1.50, lifting accuracy 10% to 69% — iamrobotbear · 2026-09-25