Entire arXiv Site Released as 16TB Dataset: 3.1M Papers on HuggingFace
secemp9 · x · 2026-09-20
The full arXiv site is now a dataset on HuggingFace: 3,148,796 papers, every version, in LaTeX, PDFs, PostScript and HTML — 16 TB total. Cost roughly $626 in AWS egress; the official S3 bucket had gaps, filled via GCS and web scraping.
Related event: Full arXiv mirror lands on Hugging Face: 3.14M papers, 16TB(8 posts)→
More from Infra
- On-device model Cactus Needle trends on Hugging Face with tool-calling support — Cactus-Compute · 2026-09-20
- Qwen3.8 Flash Next on one RTX 5090 hits 50 t/s decode via FreeToken expert caching — dir3ctly · 2026-09-20
- Are data centers dodging taxes? Tax Foundation data on $1B facilities says no — AndyMasley · 2026-09-20
- AMD carries out serious software optimizations for Kimi-K3, analyst says — AccBalanced · 2026-09-20
- RAM price jumps from $350 to $640 in months, pricing out new PCs — chrisalbon · 2026-09-20
- NVIDIA engineer breaks down why DeepSeek re-engineered V4.1 Flash for speed — thursdai_pod · 2026-09-20