How to Get Training Data as a Hobbyist: Extracting Ebooks from LibGen
theshawwn · x · 2026-08-12
Addressing how individual hobbyists can acquire LLM training data today, developer Shawn offers a practical approach:
- Data Source: Download LibGen (a shadow library), save the epub files, and discard the rest.
- Extraction Script: Run the files through his open-source epub2text script to convert them into plain text.
He notes that this is currently the only truly decentralized training data available. While acquiring them is technically illegal, no one will care unless you build a business on top of it. For legitimate commercial projects, he suggests following the lead of major AI labs by buying used physical books in bulk and digitizing them.
More from Research
- Study Finds: Higher Reasoning Effort Makes LLMs Less Likely to Ask Clarifying Questions — xuanalogue · 2026-08-12
- Open Dataset Measures AI's Actual Impact on Accelerating Scientific Discovery — soumitrashukla9 · 2026-08-12
- Six Months of AI Auto-Research Tools, But No Clear Acceleration in Algorithmic Efficiency — soumitrashukla9 · 2026-08-12
- Research Reveals the Personality Evolution of the Grok Model Family — DevDminGod · 2026-08-12
- STACX: A Modular Infrastructure for End-to-End Agentic RL — daibond_alpha · 2026-08-12
- NeurIPS 2026 Workshop: World Models for High-Stakes Healthcare — yaringal · 2026-08-12