Re-uploading a 239GB robot dataset moved only 2.6MB — inside a petabyte-scale data pipeline
mervenoyann · x · 2026-10-27
Robot data company microagi details its training-data infrastructure built with Hugging Face and Cloudflare.
- Content-addressed dedup exploits how repetitive robot recordings are: a 239GB dataset re-uploaded to HF with small changes sent only 2.6MB over the network, done in about a minute.
- Customers select data via a portal and it starts arriving in minutes to AWS, GCP, Cloudflare, Hugging Face or any cloud, with capacity of 1PB+ per day per customer (>10GB/s). Fan-out via Cloudflare Workers delivered >1PB for roughly $500.
More from Embodied
- 3am open-heart surgery on a Unitree G1 in Zurich sparks 'robot hospitals' meme — lukas_m_ziegler · 2026-10-27
- RTX Spark laptops impress analyst: 30B local model runs quiet with great battery life — Scobleizer · 2026-10-27
- ROSCon UK photo tour: BBC's whale robot, da Vinci surgical robot and the OG PR2 — kamathsblog · 2026-10-27
- Teardown of a humanoid robot Z-axis linear actuator, with Zmotor Q&A — MarwaEldiwiny · 2026-10-27
- GPT-6 Astra Beats Gemini 3.8 Flash in Paired Robot Test, Yet Neither Completes the Task — qinzytech · 2026-10-27
- Teardown of Zmotor LA500 humanoid linear actuator: 324 g, 500 N peak thrust — MarwaEldiwiny · 2026-10-27