Data preparation is architecture: a six-step pipeline shared by fine-tuning and RAG

Puzzleheaded_Box2842 · reddit · 2026-10-19

Whether you're fine-tuning or building RAG, raw data rarely arrives usable — extraction artifacts, inconsistent fields, duplicate or unsupported generated examples. The author argues much of that prep work can share one infrastructure.

A practical six-component architecture:

Normalize inputs: parse docs/JSON/JSONL/CSV into records with explicit content fields plus source identifiers.
One responsibility per operator: cleaning, chunking, generation, evaluation, filtering as replaceable steps with defined I/O.
Separate model access from logic: generation/evaluation call a serving interface (local or API) so backends swap without pipeline rewrites.
Cheap rules before model checks: drop empties, normalize whitespace, length checks, then exact/approximate dedup and model-based evaluation.
Build different outputs from one prepared base: instruction–response pairs scored for fine-tuning; cleaned passages for RAG indexing, with QA pairs reserved for eval.
Store intermediates: inspect what was removed or added and reuse earlier results when tuning later stages.

The payoff: you can change one step and see exactly which records disappeared or how a new threshold changed the dataset. Implemented in the open-source OpenDCAI/DataFlow.

Original post →

Data preparation is architecture: a six-step pipeline shared by fine-tuning and RAG

More from coding & agent