My AI news pipeline: MinHash+LSH dedup and role-based classification with JEV
Short-Balance-1542 · reddit · 2026-10-02
The author shares a hands-on pipeline for keeping up with daily AI news:
- Near-duplicate detection: MinHash + LSH to spot the same story across sources cheaply, avoiding per-article LLM comparison costs.
- Role-based classification: traditional ML and the open-source openjev underperformed; after getting off JEV's waitlist, articles are sorted into Researcher / Engineer / Startups / Noise, with encouraging results.
- Next steps: separate signals for relevance, impact, source reliability, and factual accuracy before ranking into a personalized daily email — an intelligence layer instead of raw aggregation.
Open questions: combining MinHash/LSH with embeddings, evaluating classifiers with overlapping categories, and ranking by impact at low inference cost.
More from coding & agent
- jasonkneen: every personal agent has the same unsolved problem — jasonkneen · 2026-10-03
- Dev builds WoW private server, browser client, and an MCP agent to play it — ProfessorMunchy · 2026-10-03
- Y Combinator's QM team-oriented AI agent harness is now free on Agent37 — ycombinator · 2026-10-03
- OpenAI's Dot Agent Spotted Production Outage Five Minutes Before DevDay Live Demo — lennysan · 2026-10-03
- LangChain's Chase Amps Up 'Durable Harness' Debate as Pi Durable Draws HN Praise — hwchase17 · 2026-10-03
- Open-source Jev + PageIndex searches 300-page docs without a vector DB or embeddings — gaganghotra_ · 2026-10-03