DataFlow: a pipeline/operator/prompt architecture for LLM training data prep
Puzzleheaded_Box2842 · reddit · 2026-11-17
A deep dive into the overlooked engineering between acquiring training data and actually training: turning heterogeneous sources into training examples, deciding what to keep, and inspecting each processing step.
The architecture separates workflow into pipelines (step sequences), operators (generation, refinement, evaluation, filtering), and prompts (instructions for LLM-based operators). Steps exchange structured data with explicit fields, so intermediate outputs reveal where bad examples entered or good ones were dropped.
Key argument: generating an answer doesn't prove correctness, and scoring doesn't set thresholds — keeping them separate lets you swap evaluators or filtering policies without touching generation logic. Pipelines must also reflect the training objective: pretraining filters by corpus quality, SFT extracts instruction-response pairs, reasoning data needs generation/categorization/difficulty estimation, Text-to-SQL needs schema context — a single "quality" score across all four hides real differences.
Model access is a separate component (API or local vLLM). Implemented in the open-source OpenDCAI/DataFlow (SIGMOD'27, 9k stars). Open question posed: when a filter removes much of your data, what evidence shows the remainder is actually better for training?
More from coding & agent
- "Summarize what you did in simple terms": the one AI coding prompt devs swear by — js_craft_hq · 2026-11-17
- 39 open-weight models, 11 days, 8,199 runs: 75.7% of agent failures came from runs that did call tools — Green_Competition_21 · 2026-11-17
- Gemini 4 Carbon vibe check: boldest UI designs, least reliable finishes — every · 2026-11-17
- gputaskman: an open-source htop for your GPU that breaks down VRAM by process and LLM KV-cache — alex_verem · 2026-11-17
- Matt Pocock's /constrain Idea: Cage Agents With Brutal Lint Rules — mattpocockuk · 2026-11-17
- Editing WordPress via MCP Feels Like "Superpowers"; Contributors Want It in Core — VeryWellVersed · 2026-11-17