Dev Fixes 3 Agent Eval Holes: Blind Recount Hits 89% Agreement, Pooled 4% Miss Rate Hid a Bad Lane
armanidev_ · reddit · 2026-11-23
Following up on community feedback, the author patched three holes in his agent pipeline that curates business-idea requests from Reddit posts:
- Blind recount: recount agents now see only shuffled raw titles (labels stripped), run by 3 fresh agents — they agreed with the original labels on 1,177 of 1,327 posts (89%). Surprise: two counts both totaled 10 website requests, but only 8 were the same people. Matching totals ≠ agreement.
- Miss rate by lane: a seeded audit of 50 "other" posts showed a 4% miss rate (honest range 1–13%), but both misses were video-editing posts; blind agents flagged 19 of 78 in that bucket — the pooled number was hiding a bad lane.
- Baseline: every run now logs four boring HN topics (coffee, chess, weather, bicycle) as controls; they only count after staying flat for 3+ runs (currently 1).
He ends by asking: when agents' totals match but individual calls don't, which do you trust?
More from coding & agent
- huashu-design 3.5 ships: WeChat mini-program design support, two days after 3.4 — AlchainHust · 2026-11-23
- Sharing one memory across Claude Code, Codex and local models over MCP: what worked — JoeBro121211 · 2026-11-23
- Firecrawl vs context.dev: Developers weigh web data stacks for AI agents — lewishorsley96 · 2026-11-23
- Popfold: an open-source three.js engine built with parallel Claude Code sessions — cherichita · 2026-11-23
- Weaviate Podcast #150 dives deep into rerankers and agentic search — CShorten30 · 2026-11-23
- Job-hunting agent fixes reviewed: idempotency keys help, but AUC stays 0.645 — nivokvo · 2026-11-23