DataFlow: a pipeline/operator/prompt architecture for LLM training data prep

Puzzleheaded_Box2842 · reddit · 2026-11-17

A deep dive into the overlooked engineering between acquiring training data and actually training: turning heterogeneous sources into training examples, deciding what to keep, and inspecting each processing step.

The architecture separates workflow into pipelines (step sequences), operators (generation, refinement, evaluation, filtering), and prompts (instructions for LLM-based operators). Steps exchange structured data with explicit fields, so intermediate outputs reveal where bad examples entered or good ones were dropped.

Key argument: generating an answer doesn't prove correctness, and scoring doesn't set thresholds — keeping them separate lets you swap evaluators or filtering policies without touching generation logic. Pipelines must also reflect the training objective: pretraining filters by corpus quality, SFT extracts instruction-response pairs, reasoning data needs generation/categorization/difficulty estimation, Text-to-SQL needs schema context — a single "quality" score across all four hides real differences.

Model access is a separate component (API or local vLLM). Implemented in the open-source OpenDCAI/DataFlow (SIGMOD'27, 9k stars). Open question posed: when a filter removes much of your data, what evidence shows the remainder is actually better for training?

Original post →

More from coding & agent

coding & agent channel →