Open reproduction of Meta’s REWIRE data pipeline cuts the cost to about $11
vanstriendaniel · x · 2026-07-21
A small open re-run of Meta’s REWIRE pipeline can reproduce the dataset far more cheaply
A proof-of-concept reproduction of Meta/FAIR’s REWIRE (“Recycling the Web”) shows how discarded web pages can be rewritten by an LLM and re-scored by the same quality filter to produce pre-training data.
- The original paper used Llama-3.3-70B as the rewriting teacher.
- This re-run used Hugging Face Jobs and datatrove, and the author says the whole proof of concept cost about $11 in GPU time.
- In the experiment, a 9B model beat the paper’s 70B teacher on the paper’s own gate:
- 61.7% of rewrites passed the quality filter vs 49.6% for the 70B teacher.
- Blind paired judges preferred the 9B rewrites 20–4.
- Cost-wise, the 9B arm came out to about $0.60 for 600 docs, or roughly $0.64 per million kept tokens.
- The author extrapolates that Meta’s released 44.4B-token set would cost about $28k to reproduce at the 9B rate, versus $369k at the 70B rate.
The repo is fully open, including the pipeline, the 1,953 rewrites, receipts, and a small CPU script that re-scores every rewrite.
More from Infra
- Devin adds e2b sandboxes for remote agent execution — badphilosopher · 2026-07-22
- Arbitrum fee simulation shows higher gas capacity but lower L2 revenue under ArbOS61 — tomwanhh · 2026-07-22
- NVIDIA pushes OpenUSD as the common layer for simulation and physical AI — MonaJalal_ · 2026-07-22
- SkyPilot exits stealth with $20M to unify fragmented GPU compute across five clouds — skypilot_org · 2026-07-22
- Production AI budgets include retries, routing, caching and observability—not just token prices — arx-go · 2026-07-22
- NVIDIA briefs analysts on Vera CPU and doubles down on monolithic agentic design — BenBajarin · 2026-07-22