Turning newsletter edits into AI checks lifts frontier models 25+ points on summaries
danshipper · x · 2026-11-16
Dan Shipper of Every describes a workflow for encoding editorial taste into evaluable AI checks: real Context Window drafts are saved alongside his actual edits and reactions in Every Checks, with each draft-vs-gold-case difference converted into a binary yes/no check. Eight real cases yielded 74 checks defining what good writing looks like.
The team then built a reusable short-idea-summary skill and iterated it from v1 to v9, testing each version in isolated zero-context sandboxes across four frontier models (Gemini 4 Hydrogen, Claude Opus 5.5, GPT-6 Astra, Claude Sonnet 5.5, all High effort), blind-graded by claude-sonnet-5-5 against the 74 checks.
- v1: all models scored 34%–56% (Opus 5.5 56%, GPT-6 Astra 46%, Gemini 4 Hydrogen 40%, Sonnet 5.5 34%)
- v9: models converged to 75%–81%, led by Gemini 4 Hydrogen at 60/74 (81.1%); Opus 5.5 and GPT-6 Astra at 58/74; Sonnet 5.5 at 56/74
Takeaway: collect gold cases, turn their differences from mediocre drafts into checks, then optimize a skill against those checks—turning daily editorial feedback into a compounding asset applicable to any quality-critical writing task.
More from coding & agent
- levelsio runs every business on VPS servers managed solely via SSH and Claude Code — alvelda · 2026-11-16
- Dev builds remote MCP server with inline MCP Apps video player for 5,500 meme clips — Mediocre-Number-5498 · 2026-11-16
- Developer's autonomous "factory" builds usable app for ~$40 in Claude credits — threepointone · 2026-11-16
- 20-year AI veteran curates 19 free open-source repos for building AI agents — MaryamMiradi · 2026-11-16
- Local agent testing: model choice beats harness, coding-focused harnesses win across tasks — dh7net · 2026-11-16
- KubeCon's big theme is inference: a full-stack breakdown from DRA GPU slicing to vLLM and llm-d — AccBalanced · 2026-11-16