Open reproduction of Meta’s REWIRE data pipeline cuts the cost to about $11
vanstriendaniel · x · 2026-07-21
A small open re-run of Meta’s REWIRE pipeline can reproduce the dataset far more cheaply
A proof-of-concept reproduction of Meta/FAIR’s REWIRE (“Recycling the Web”) shows how discarded web pages can be rewritten by an LLM and re-scored by the same quality filter to produce pre-training data.
- The original paper used Llama-3.3-70B as the rewriting teacher.
- This re-run used Hugging Face Jobs and datatrove, and the author says the whole proof of concept cost about $11 in GPU time.
- In the experiment, a 9B model beat the paper’s 70B teacher on the paper’s own gate:
- 61.7% of rewrites passed the quality filter vs 49.6% for the 70B teacher.
- Blind paired judges preferred the 9B rewrites 20–4.
- Cost-wise, the 9B arm came out to about $0.60 for 600 docs, or roughly $0.64 per million kept tokens.
- The author extrapolates that Meta’s released 44.4B-token set would cost about $28k to reproduce at the 9B rate, versus $369k at the 70B rate.
The repo is fully open, including the pipeline, the 1,953 rewrites, receipts, and a small CPU script that re-scores every rewrite.
More from Infra
- DeepSpeed shows AutoEP/AutoTP/AutoSP parallelism gains, hosts office hours — PyTorch · 2026-10-27
- Cloudflare Durable Objects validated as top architecture for AI agent WebSocket servers — tristanbob · 2026-10-27
- Gary Marcus: With Q3 data added, Anthropic's compute-efficiency growth shows deceleration — GaryMarcus · 2026-10-27
- SpaceX locks in $40B GPU financing deal with Apollo, the largest ever committed — TechCrunch AI · 2026-10-27
- Inference is just Kubernetes with a multi-trillion-dollar market, and the communities overlap — matt_slotnick · 2026-10-27
- vLLM eyes Rust migration as inference engines race to rewrite the stack — AccBalanced · 2026-10-27