Entire arXiv released as a HuggingFace dataset: 3.1M papers, 16TB in LaTeX, PDF, HTML
secemp9 · x · 2026-09-20
A developer has packaged the entire arXiv site as a public dataset on Hugging Face: 3,148,796 papers with every version included, in LaTeX, PDF, PostScript, and HTML — 16 TB in total. The author notes dedup was version-based, so counting versions brings the total to over 6M documents. A rare full-corpus, multi-format resource for large-scale training, retrieval, or literature analysis.
Related event: Complete arXiv Mirror Lands on Hugging Face: 3.14M Papers, 16TB(9 posts)→
More from Research
- Gradient-free weight editing cuts VLM FigStep attack success from 49% to 2% — lateinteraction · 2026-11-04
- Anthropic says it may imminently hit RSP threshold for AI-accelerated R&D, releases provable inference work — tobyordoxford · 2026-11-04
- Google scanned 1,033 real household products, packed into a 345 MB GLB for three.js — victormustar · 2026-11-04
- These 10 GPU optimization papers are why your model runs fast today — NandoDF · 2026-11-04
- INT8 isn't the problem: I benchmarked samplers on INT8 flow models — YakovSava · 2026-11-04
- Karpathy Boosts Open Hacker News Dataset: 10M+ Posts Usable as a Personal Knowledge Base — huangyun_122 · 2026-11-04