WaterCrawl: open-source crawler that turns web content into LLM-ready data
tom_doerr · x · 2026-08-27
WaterCrawl (2k stars, 252 forks on GitHub) is a web crawling application built with Python, Django, Scrapy and Celery, aimed at extracting web content and transforming it into LLM-ready data structures.
It ships with a quick Docker setup: clone the repo, copy .env.example, run docker compose up -d, and access it at localhost. When deploying on a domain or IP other than localhost, you must update the MinIO configuration in .env accordingly. The repo includes full frontend/backend, docs and tutorials.
More from coding & agent
- Solana agent swarm developed private vocabulary and message board logic — tszzl · 2026-08-27
- SiteSpeakAI Launches Google Chat Integration for Internal Knowledge Base Bots — hermanschutte · 2026-08-27
- Apodex-1.1-mini: Qwen3.5-MoE Powered Multimodal Agent Model — apodex · 2026-08-27
- RubSE: Self-Evolving UI-to-Code Generation via Rubrics — Tianyi Xiong · 2026-08-27
- Weeks-long Sol agent swarm burned billions of tokens and invented its own private vocabulary — Sauers_ · 2026-08-27
- Memoria V4.5: 82.6% Recall on 4GB RAM CPU-Only Machine — kitkatz69 · 2026-08-27