Spark creator Josh Rosen: AI systems should steal far more from data engineering
colinmcnamara · x · 2026-10-06
Apache Spark co-creator Josh Rosen argues AI systems should borrow more from decades of data engineering practice: use DAGs instead of one giant agent loop; track lineage of sources, artifacts, prompts, and model versions; do change detection to know which generated artifacts are stale; apply incremental processing instead of rerunning whole agents; and treat context handling as classic ETL. These problems have long been solved — AI engineering should reuse the solutions.
More from coding & agent
- The handoff test: approve, revoke, transfer to a fresh agent—does human authority survive? — tallmetommy · 2026-10-06
- Beam agent one-shots a full end-to-end Unsloth training pipeline in OpenCode with a single prompt — bhutanisanyam1 · 2026-10-06
- Open-source 'Clay killer' launched: 85% cheaper, top people-search accuracy, 25x faster — Scobleizer · 2026-10-06
- New Obsidian plugin qiaomu-ui-learn trains your vibe-coding UI taste with copyable prompts — vista8 · 2026-10-06
- Developer open-sources Semantica, a Palantir-style enterprise BI alternative (13.7k stars) — mdancho84 · 2026-10-06
- Hands-on: Orchestrator splits big features into PRs, Orca handles frontend work — julianweisser · 2026-10-06