Micron and Meta paper says Spark can slow down 38× when shuffle spills to SSD

dr_alphalyrae · x · 2026-07-27

Micron/Meta white paper shows DRAM shortage can cause 38× slowdowns in Spark workloads

A joint Micron/Meta paper says Spark workloads hit a major memory wall when shuffle data no longer fits in DRAM and spills to SSD.

• In the standard workload, the shuffle-heavy stage becomes the bottleneck and shows a 38× slowdown once it spills to disk.
• Under a heavier 3× workload, the bottleneck shifts to scan-stage rereads, and a smaller OS page cache causes repeated SSD rereads.
• The paper argues that larger memory capacity is doing two jobs at once: keeping the shuffle working set in DRAM and preserving a big enough page cache.
• The takeaway is that provisioning enough DRAM for both shuffle partitions and page cache is essential to avoid order-of-magnitude slowdowns from disk I/O.

Original post →

More from Infra

• NVIDIA says AdamW hits a scale ceiling as SOAP and Muon beat it on trillion-token runs — omarsar0 · 2026-07-27
• Local LLM builders ask whether RTX Ada workstation cards are worth tracking — egudegi · 2026-07-27
• Guide breaks down how to cut Microsoft Fabric capacity costs on Azure — adnan_hashmi · 2026-07-27
• Introductory guide explains the `sempy.fabric` package for Microsoft Fabric — adnan_hashmi · 2026-07-27
• Voice AI’s real call cost is more than minutes: STT, TTS, SIP and retries — decant338 · 2026-07-27
• Inference businesses may be charging 7× to 15× more than renting a GPU — JoshPurtell · 2026-07-27

Infra channel →