Microsoft Fabric's Efficient Scaledown Cuts Shuffle-Heavy Spark Costs by 43%
adnan_hashmi · x · 2026-09-25
Microsoft shipped Efficient Scaledown in preview for Spark on Fabric, targeting the classic problem of slow cluster scale-down after shuffle-heavy jobs.
- Shuffle data lives on executor-local disks that die with the executor, so the autoscaler can't safely release nodes still holding blocks — creating "zombie executors" that idle and bill
- Efficient Scaledown decouples shuffle from executor lifetime; Microsoft's benchmark shows 43% less compute
- No code changes required; relevant for GROUP BY/JOIN-heavy workloads
Now available in preview.
More from Infra
- UK Sovereign AI invests in Fractile to build next-gen frontier AI chips — charlieharris01 · 2026-11-11
- What happened when I moved vector retrieval into Snowflake SQL instead of orchestrating RAG outside — dima806_dima · 2026-11-11
- AI chip startup Fractile raises $720M Series C for DRAM-on-processor design — Tom_Westgarth15 · 2026-11-11
- Mobile AI Factory: agentic pipeline builds on-device SLMs that beat 5-shot frontier models — kalyan_kpl · 2026-11-11
- Vivo's Starlink direct-to-cell service hits 10M users in two weeks across Brazil — XFreeze · 2026-11-11
- Nan Ya Plastics to spend NT$11.53B on fifth glass fabric plant on AI server demand — zephyr_z9 · 2026-11-11