Entire arXiv site released as 16TB Hugging Face dataset: 3.1M papers, all versions
MaziyarPanahi · x · 2026-09-20
A new dataset on Hugging Face covers the entire arXiv site: 3,148,796 papers with every historical version included, shipped in LaTeX, PDF, PostScript, and HTML formats — about 16 TB in total. Commenters joked about the scale, noting you'd practically need a cluster (or GPU-accelerated tooling) to compile that much LaTeX, and thanked the maintainers for the effort.
Related event: Complete arXiv Mirror Lands on Hugging Face: 3.14M Papers, 16TB(9 posts)→
More from Research
- Anthropic says it may imminently hit RSP threshold for AI-accelerated R&D, releases provable inference work — tobyordoxford · 2026-11-04
- Google scanned 1,033 real household products, packed into a 345 MB GLB for three.js — victormustar · 2026-11-04
- These 10 GPU optimization papers are why your model runs fast today — NandoDF · 2026-11-04
- INT8 isn't the problem: I benchmarked samplers on INT8 flow models — YakovSava · 2026-11-04
- Karpathy Boosts Open Hacker News Dataset: 10M+ Posts Usable as a Personal Knowledge Base — huangyun_122 · 2026-11-04
- PixelCraft: A Multi-Agent Framework Targets Multimodal AI's Chart-Reading Failures — burkov · 2026-11-04