Archil claims self-hosted Spark on its S3-as-local-disk storage runs 2-5x faster than Databricks for under half the cost
massimosgrelli · x · 2026-11-10
jhleath follows up his open-source ClickHouse cost-cutting pitch with a Spark claim: self-hosting Apache Spark on Archil storage runs 2-5x faster than Databricks serverless for under half the cost.
- Argument: commercial open source vendors ship a hobbled OSS version (e.g. ClickHouse OSS needs local disks, no S3 support) because self-hosting the data infrastructure was genuinely hard
- Archil's fix: use S3 as a local disk, providing high-performance shared storage across nodes
- The Spark guide runs a 4-node, 32-vCPU-each cluster that starts on demand (1s cold start, ready 5s after script start), reads Parquet from S3 in place, and writes Delta Lake tables — whole job about 30 seconds once cached
- Vendor-claimed numbers in a marketing-flavored infra pitch
More from Infra
- Seroter's daily reading list: new TPUs, Claude 5.2 price cut, and the future of the PM role — rseroter · 2026-11-10
- DOE Funds $11.5M GridFM 2.0 AI Project Targeting 1B Grid Scenarios in 24 Hours — ingliguori · 2026-11-10
- NVIDIA open-sources Molt: a 9k-line RL framework where the whole setup is one Python file — algo_diver · 2026-11-10
- Microsoft Foundry adds Kling, LTX, and MiniMax video models via new fal partnership — gorkem · 2026-11-10
- Uno paper: diffusion adapters on AR LLMs beat speculative decoding with lossless speedups — LucaAmb · 2026-11-10
- Norway Suspends New Data Center Electricity Reservations Until as Late as August 2027 — Polymarket · 2026-11-10