Complete arXiv Mirror Lands on Hugging Face: 3.14M Papers, 16TB

On Sept 20, developer secemp9 published the dataset secemp9/arxiv-complete on Hugging Face, turning the entire arXiv site into a public dataset. It is a rare full-mirror, all-versions, multi-format arXiv release that significantly lowers the barrier to large-scale paper corpora, but it is self-funded, so sustainability depends on community support.

Confirmed

• The dataset contains 3,148,796 papers covering every version, totaling roughly 16TB
• Multiple formats provided: LaTeX source, PDF, PostScript and HTML, stored as parquet and tagged for text-generation and text-retrieval tasks
• The dataset reached Hugging Face's trending list after release
• Hosting costs about $249/month ($3,000/year); pulling data from AWS earlier incurred about $626 in egress fees
• secemp9 found the official arXiv S3 bucket incomplete and had to fill gaps from other sources such as GCS
• The release was widely shared: MaziyarPanahi promoted it multiple times, and burnytech called it a stunning pretraining corpus

Why it matters

• A complete, all-versions, multi-format public arXiv dataset greatly reduces friction for researchers needing large paper corpora
• An independent developer is bearing bandwidth and storage costs alone; long-term maintenance relies on community support, and he has opened a donation page

2026-09-20 ~ 2026-09-21 · 9 related posts

• [source] Entire arXiv uploaded to Hugging Face: 3.15M papers, every version, 16TB in LaTeX/PDF/HTML — secemp9 · 2026-09-20
• Entire arXiv Site Released as 16TB Dataset: 3.1M Papers on HuggingFace — secemp9 · 2026-09-20
• [source] Dev pays $249/month to host arXiv dataset mirror, opens donations to cover costs — secemp9 · 2026-09-20
• arxiv-complete: a 100M+ row arXiv dataset trends on Hugging Face — secemp9 · 2026-09-20
• Entire arXiv released as a 16TB HuggingFace dataset: 3.1M papers, every version — udmrzn · 2026-09-21

4 near-duplicate retellings: secemp9 · MaziyarPanahi · MaziyarPanahi · burny_tech