Running Data Pipelines Cheaply with HF Jobs
vanstriendaniel · x · 2026-07-16
The author explains how they processed 1.2 million Common Crawl pages for under $1 without needing a Slurm cluster.\n\nThe core approach involves connecting a DataTrove pipeline to Hugging Face Jobs:\n- Reusing the same pipeline code with a different executor\n- Distributing tasks to cloud Jobs for parallel execution\n- Chaining CPU filtering and GPU inference within the same workflow\n- Automatically skipping completed tasks during reruns\n\nThis process turned 1.2 million pages into 228,000 clean documents. The author notes that while JobsPipelineExecutor is currently experimental, it has already been merged into DataTrove main.
Related event: Developer Processes 1.2M Web Pages for Under $1 Using HF Jobs(3 posts)→
More from Infra
- Nebius says SlimSpec speeds speculative decoding 8–9% without shrinking the vocabulary — Arindam_1729 · 2026-07-21
- NVIDIA brings its Cosmos 3 Edge world model to Jetson for on-device robot control — liu_mingyu · 2026-07-21
- A silicon photonic reservoir chip compensates fiber distortion in real time at 28 Gbps — bravo_abad · 2026-07-21
- Chamath says open-sourcing Grok would push AI margins from models to infra and apps — Dan_Jeffries1 · 2026-07-21
- EU AI competitiveness is under pressure as firms double down on chips, ethics, and talent — nordicinst · 2026-07-21
- AI bottlenecks are shifting to memory, optics, yield control and power — thedealdirector · 2026-07-21