I pointed Claude at 11 million news articles — each event averages 12 near-duplicate reports
conurbano · reddit · 2026-10-06
The author runs a news-clustering side project for agents: 100k articles/day, embedded, clustered into events, LLM-summarized, and linked into timeline "situations". After 164 days, a fresh Claude Code session analyzed a read-only copy of the database.
Key findings:
- 11,079,667 distinct articles became 941,096 event clusters — 12 articles per real-world event;
- only 1.2% of clusters keep gaining coverage past day 3; token waste is mostly day-one rewrites;
- 144,891 situations tracked, 64% still developing after a week;
- Trump is the most-mentioned person, more than the next five combined.
Metrics and methodology are published as The Observatory, and the pipeline is exposed as an MCP server at mcp.clstr.news/mcp.
More from coding & agent
- Dev returns to Codex after a week on Opus 5.5: "basically unusable" — jacob_posel · 2026-10-06
- Zip Rebuilt Its Agent on LangGraph and LangSmith, Cutting Weeks-Long Feature Work — LangChain · 2026-10-06
- Dev shows custom voxel agent converting real buildings with high detail — Promptmethus · 2026-10-06
- Cursor SDK adds run.steer() to redirect agents mid-run, backgrounding subagents — tetsuoai · 2026-10-06
- 4DCodeBench: new benchmark tests coding agents on reconstructing dynamic 3D scenes from video — CSProfKGD · 2026-10-06
- Agent harness tier list sparks debate: custom compaction harnesses should be discounted — pvncher · 2026-10-06