Spark Co-creator: AI Systems Should Borrow from Data Engineering

Apache Spark co-creator Josh Rosen argues that building AI systems should draw on decades of data engineering wisdom, including DAGs, data lineage, and incremental processing rather than monolithic agent loops.

2026-10-06 ~ 2026-10-07 · 2 related posts

• Spark creator Josh Rosen: AI systems should steal far more from data engineering — colinmcnamara · 2026-10-06
• Steal from data engineering: DAGs, lineage and incremental processing for AI agent systems — LangChain · 2026-10-07