Entire arXiv uploaded to Hugging Face: 3.15M papers, every version, 16TB in LaTeX/PDF/HTML
secemp9 · x · 2026-09-20
User secemp9 released secemp9/arxiv-complete on Hugging Face: the entire arXiv site as a dataset with 3,148,796 papers covering every version, 16 TB total.
- All formats: LaTeX sources, PDFs, PostScript, and HTML
- Structured subsets: files (54.6M rows), latex, metadata (3.15M), papertext, source, versions — auto-converted to Parquet, queryable via SQL/Dask/Polars
- License: mixed-arxiv-author-licenses (check per-paper copyright)
- A heavyweight resource for text generation and retrieval work
Related event: Complete arXiv Mirror Lands on Hugging Face: 3.14M Papers, 16TB(9 posts)→
More from Infra
- Malaysia poised to become a compute superpower with both US and Chinese chips by next year — basedjensen · 2026-11-04
- ByteDance details USDB storage system: 85.8% lower write latency for recommender sequences — _reachsumit · 2026-11-04
- Oracle cancels $7B chip leasing deal with Tencent, FT reports — SumitGup · 2026-11-04
- Inference engine shrunk to 1023 bytes, still runs SmolLM2-135M — cneuralnetwork · 2026-11-04
- Musk says space data centers are 'looking pretty appealing right now' — elonmusk · 2026-11-04
- Data centers are turning to old jet engines as the wait for power grows — IndependentCoast3974 · 2026-11-04