Solo dev runs 6,000+ unattended news episodes in 19 months — eval loops, not models
jwdNyc · reddit · 2026-10-21
A solo developer shares a fully automated news video pipeline: 6,000+ episodes in 19 months (3/day on a 6-hour cadence) at $3–3.50 per episode in inference. The system ingests newsletters, decides if there's enough for an episode, writes the script, builds an avatar video in a cloned voice, and publishes video, blog, and LinkedIn posts.
The core lesson: the model is swappable; the eval loops are the system. Key practices:
- Evals must block, not report: every output is checked against required/prohibited behaviors; missing behavior = no publish.
- Database beats prompt for dedup: a Postgres topic store queried before writing; prompts drift, queries don't.
- Cross-review architecture with a different model: one writes the plan, another critiques it — this surfaced exposed API keys.
- Deterministic where correctness matters: thumbnail typography is never left to the model.
- Write handoffs, not long contexts: small work units with written handoffs, compact before switching tools.
Where it breaks: distribution — 6,400 episodes, under 250 subscribers.
More from coding & agent
- 90% of big-company knowledge lives in heads, and "company brain" startups are in for pain — gabriel1 · 2026-10-21
- EAU: Co-evolving user simulator, environment, and RL setup for training LLM agents — UAlberta · 2026-10-21
- Meta's Self-Credit: Dense Rewards from Sparse RLVR via Intra-group Rescaling — meta · 2026-10-21
- Notion details its Auto model router: Elo-based evals, 62-93% cheaper than flagship models — ivanhzhao · 2026-10-21
- Dev sets up automation so agents proactively pick up stale issues and PRs — vikvang1 · 2026-10-21
- Tamper-proof agent logs still can't prove the agent did the job: who reads the record back? — ZealousidealAct9661 · 2026-10-21