Microsoft Fabric's Efficient Scaledown Cuts Shuffle-Heavy Spark Costs by 43%
adnan_hashmi · x · 2026-09-25
Microsoft shipped Efficient Scaledown in preview for Spark on Fabric, targeting the classic problem of slow cluster scale-down after shuffle-heavy jobs.
- Shuffle data lives on executor-local disks that die with the executor, so the autoscaler can't safely release nodes still holding blocks — creating "zombie executors" that idle and bill
- Efficient Scaledown decouples shuffle from executor lifetime; Microsoft's benchmark shows 43% less compute
- No code changes required; relevant for GROUP BY/JOIN-heavy workloads
Now available in preview.
More from Infra
- How GPUs really run deep learning: a primer on memory hierarchy and optimization — goyal__pramod · 2026-09-25
- VeriTile embeds Triton GPU kernels in Lean, with AI agents writing machine-checked correctness proofs — KaiyuYang4 · 2026-09-25
- Merge Gateway Launches Batch Inference at ~50% of Standard Prices — shensi · 2026-09-25
- New deep-dive article on scaling LLM inference in production — abhijithneil · 2026-09-25
- Lambda engineer shares local inference build rule: 27B models need 24-32GB VRAM — TheZachMueller · 2026-09-25
- Pokee AI demos 36B agent model running fully local on Snapdragon X2 Elite with 32GB RAM — Kyrannio · 2026-09-25