Kernel-by-kernel: why PyTorch beat JAX on the same Noise benchmark recipe
RobertTLange · x · 2026-10-21
JAX veteran RobertTLange (author of evosax) revisits the JAX vs PyTorch debate with a first-hand experiment.
- Context: a longtime JAX fan—jit + v/pmap made evolution strategies feel magical for whole-population evaluation. He set a record (#29: 975 updates on 8×H100, val loss < 4.2) on the Noise benchmark's zeroth-order track using JAX + Pallas kernels.
- Key finding: maintainers ported his entry to PyTorch with the same estimator, recipe, work split, and collectives—and it ran faster: 6.35s vs 6.8–7.6s for JAX on the same node. The benchmark now requires PyTorch for new entries.
- He then profiled both kernel by kernel: on the GPU they're essentially the same program; a typical update takes 6.49ms in PyTorch, nearly identical to JAX, with differences coming from framework-level overhead.
More from Infra
- CoreWeave launches Physical AI Launchables: run robotics and simulation workloads with one command — ayushthakur0 · 2026-10-21
- NVIDIA locks up massive ABF substrate capacity again, cementing its win in the AI compute cycle — BenBajarin · 2026-10-21
- php2shell Flaws Hit ~70% of Websites; Cloudflare Shipped WAF Rules 9 Hours Early — Cloudflare Blog · 2026-10-21
- SemiAnalysis: 1,730 solar pullers yield ~30 GWp, only 4-7 GW of firm datacenter power — BenBajarin · 2026-10-21
- Frozen 16 MB vocabulary slashes small LLM size to 46 MB with no quality loss — pmttyji · 2026-10-21
- llampart v2: open-source local LLM chat UI with charts, Mermaid and MCP support — mossy_troll_84 · 2026-10-21