Entire arXiv Site Released as 16TB Dataset: 3.1M Papers on HuggingFace
secemp9 · x · 2026-09-20
The full arXiv site is now a dataset on HuggingFace: 3,148,796 papers, every version, in LaTeX, PDFs, PostScript and HTML — 16 TB total. Cost roughly $626 in AWS egress; the official S3 bucket had gaps, filled via GCS and web scraping.
Related event: Complete arXiv Mirror Lands on Hugging Face: 3.14M Papers, 16TB(9 posts)→
More from Infra
- These 10 GPU optimization papers are why your model runs fast today — NandoDF · 2026-11-04
- One prompt to audit cross-region traffic: dev cut trans-Atlantic latency by moving a service — DanielLockyer · 2026-11-04
- SurrealDB 4 launches as multi-model ACID database for the agentic era — Scobleizer · 2026-11-04
- Fire at Taiwan Panasonic PCB materials plant may tighten CCL supply further — zephyr_z9 · 2026-11-04
- Same Claude model served by Bedrock one day and Anthropic the next, dev finds — georgos_pappas · 2026-11-04
- Self-improving software is inevitable: closing the agent-observability loop — bibryam · 2026-11-04