Trafilatura: The Open-Source Tool Cleaning the Web for AI Training
alex_verem · x · 2026-08-19
Trafilatura is an open-source Python tool that strips raw HTML down to just the main content and metadata (title, author, date), outputting clean text, Markdown, or JSON. It powers major datasets like FineWeb and RefinedWeb and is widely used by top labs for RAG preprocessing. Maintained by a single researcher, it consistently ranks at the top in independent benchmarks for content extraction and is completely free to use without an API key.
More from Infra
- Polymarket: 70% chance a state enacts data center moratorium by 2026 — Polymarket · 2026-08-20
- Virginia County Expects $1.3B in Data Center Tax Revenue — Polymarket · 2026-08-20
- DeepSeek price hike sparks local setup: hybrid routing cost tactics — LunarLurker-42 · 2026-08-20
- Zero-Markup API Gateway Secures $13.3M in 48 Hours, Exposing Hidden AI Costs — Scobleizer · 2026-08-20
- OpenWolf: Optimize AI Coding Assistant Context with 7 Invisible Hooks — tom_doerr · 2026-08-19
- Gatana adds encrypted persistent storage to MCP Gateway — Gatana_Official · 2026-08-19