DataFlow-Harness: A Grounded Code-Agent Platform for Constructing Editable LLM Data Pipelines
Runming He, Zhen Hao Wong, Hao Liang, Zimo Meng, Chengyu Shen, Xiaochen Ma, Wentao Zhang
cs.SE, cs.AI
2026-07-18
Coding agents produce disposable data-processing scripts; DataFlow-Harness grounds them through MCP into typed incremental mutations against a live pipeline backend, producing editable pipelines at 93.3% end-to-end pass and about 73% lower cost than a vanilla agent.
LLMs increasingly automate data-processing workflows, but coding agents typically produce scripts that are not materialized as persistent, editable platform artifacts. The paper calls this the NL2Pipeline gap: users state needs in natural language, while production needs structured, persistent, visualizable, editable, reusable pipeline assets. Throwaway scripts live only as source code, cannot be audited in a graphical workflow UI, hallucinate unavailable operators or outdated platform assumptions, and cannot integrate with governance. The agent looks like it succeeded, but the artifact is unusable.
Four components.
Data Pipeline Backend represents pipelines as DAGs with typed mutations. The agent is constrained to typed incremental mutations, never free-form scripts, and every change must pass validation: the result must remain a DAG and adjacent operator schemas must be compatible. One persistent representation is shared across the conversational and visual interfaces.
MCP Tools Layer exposes the live operator registry and current pipeline state to the agent via a Request-Validate-Commit protocol: the agent fetches the latest state each turn, issues structured mutations, passes structural and schema checks, then commits and broadcasts over WebSocket. The agent operates against live platform semantics rather than hallucinated or stale assumptions.
DataFlow-Skills encodes domain construction knowledge that raw operator specs cannot: procedural blueprints give the recommended build order (schema inference, operator selection, parameter config, serving verification); compositional constraints give compatibility rules (modality matching, field flow). Without Skills the agent may pick wrong operators, skip prerequisites, or build structurally valid but semantically wrong pipelines.
DataFlow-WebUI is a dual-modal interface: conversational authoring injects pipeline state and the operator registry into Claude Code via MCP, and a visual DAG editor lets users inspect changes, tune parameters, relink edges, and add or remove operators. Manual edits commit directly to the backend so the next agent turn operates on the latest state.
Twelve tasks across six data-processing scenarios, ten trials each (120 runs), on Claude Opus 4.7:
| Method | Artifact | End-to-end pass | Cost | Latency |
| Vanilla CC | throwaway script | 91.7% | $0.950 | 190.7s |
| MCP-only | native DAG | 83.3% | $0.321 | 105.5s |
| DataFlow-Harness | native DAG | 93.3% | $0.261 | 95.5s |
The key finding: MCP-only (83.3%) is actually lower than vanilla scripting (91.7%), which quantifies the NL2Pipeline gap; DataFlow-Harness closes it, ten points above MCP-only and roughly even with vanilla scripting while cutting cost about 73% and latency about 50%. Pipeline representations are far more compact than executable code, and procedural guidance streamlines construction.
Downstream training utility: a DataFlow-Harness math-reasoning pipeline trained Qwen2.5-32B-Instruct for one epoch, lifting AIME24@32 from 25.1 to 35.9 and AIME25@32 from 21.6 to 34.5; a general SFT pipeline on Qwen2.5-7B-Base raised a nine-benchmark average from 61.5 to 63.8, with MBPP 64.6 to 75.4.
The NL2Pipeline gap is a real problem for teams building agents plus data platforms. The transferable lesson is: do not let the agent emit code directly; have it make typed mutations against a live backend and feed it domain procedural knowledge. The MCP, Skills, and backend layering is worth copying for in-house data platforms.
The evaluation is narrow: one agent, one model family, a small platform-specific benchmark. The ablation does not isolate every component. Schema validation ensures structural but not semantic correctness. The paper reports averages without task-clustered confidence intervals or a pre-specified non-inferiority test, and the authors say not to read it as a general causal estimate. Cost reporting must be recomputed under prompt caching. Downstream utility rests on two case studies without multiple pipelines or seeds. Governance claims (persistence, reuse, provenance, concurrent editing, recovery) are not directly evaluated. The design is solid; the evidence is preliminary.