Open reproduction of Meta’s REWIRE data pipeline cuts the cost to about $11

vanstriendaniel · x · 2026-07-21

A small open re-run of Meta’s REWIRE pipeline can reproduce the dataset far more cheaply

A proof-of-concept reproduction of Meta/FAIR’s REWIRE (“Recycling the Web”) shows how discarded web pages can be rewritten by an LLM and re-scored by the same quality filter to produce pre-training data.

The original paper used Llama-3.3-70B as the rewriting teacher.
This re-run used Hugging Face Jobs and datatrove, and the author says the whole proof of concept cost about $11 in GPU time.
In the experiment, a 9B model beat the paper’s 70B teacher on the paper’s own gate:
61.7% of rewrites passed the quality filter vs 49.6% for the 70B teacher.
Blind paired judges preferred the 9B rewrites 20–4.
Cost-wise, the 9B arm came out to about $0.60 for 600 docs, or roughly $0.64 per million kept tokens.
The author extrapolates that Meta’s released 44.4B-token set would cost about $28k to reproduce at the 9B rate, versus $369k at the 70B rate.

The repo is fully open, including the pipeline, the 1,953 rewrites, receipts, and a small CPU script that re-scores every rewrite.

Original post →

Open reproduction of Meta’s REWIRE data pipeline cuts the cost to about $11

More from Infra