Apache Spark 4.3 ships with real-time streaming aggregation, Auto CDC and quarterly release cadence
matei_zaharia · x · 2026-11-06
Databricks announced Apache Spark 4.3, available in Databricks Runtime 19, alongside a shift to a quarterly feature-release cadence. Highlights:
- Stateful Real-Time Mode: streaming aggregation and transformWithState
- SCD Type 2 Auto CDC for history-preserving pipelines
- Project Feather for faster small-data iteration
- Expanded Spark Connect capabilities
- SQL improvements across the board
More from Infra
- Veteran investor: memory stocks aren't cheap even at P/E of 3 as LTAs are meaningless — bookwormengr · 2026-11-06
- Enterprise AI's future is local: tactical workloads favor Dell, HPE, Lenovo, Supermicro over frontier labs — DavidLinthicum · 2026-11-06
- Texas Governor-Elect: Approved Data Centers Can Be Completed, New Projects to Pay Taxes — GavinSBaker · 2026-11-06
- Marin's open 535B-A23B MoE finishes main pretraining on 18T tokens, checkpoints soon — dlwh · 2026-11-06
- Buoyant Open-Sources Kinference, an Inference Control Plane for vLLM on Kubernetes — vllm_project · 2026-11-06
- vLLM ranks #2 most-engaged open-source repo in GitHub Octoverse, #8 by contributor growth — vllm_project · 2026-11-06