Cutting AI verifier reading cost: top-50 retrieval kept just 2 of 8 minority evidence items
iMiguelmars · reddit · 2026-09-15
The author's team shares hard lessons from testing an AI verification pipeline where the bottleneck is no longer retrieval but how much evidence the system must read to be confident nothing was missed:
- Early stopping: in frozen replay cases, full recall required reading 290–426 candidates out of pools of 407–454 — far deeper than expected.
- Minority evidence: items noticed by only one of eight independent readers survived at most 2/8 at k=50; shallow cutoffs systematically drop the long tail.
- Rule-based slice skipping: one version skipped 17 slices that later turned out to contain relevant evidence, so it can't be treated as a safe filter.
- Dedup underdelivers: removing byte-identical repetition cut slice volume by only 3.3% and didn't reduce reading sessions.
The open problem: read less, keep high recall, don't disproportionately lose weird evidence, and never silently convert "not inspected" into "nothing there" — plus how to prove skipped content was actually safe to skip.
More from Research
- Microsoft's ESRL boosts MoE RL via expert-space exploration — MicrosoftResearch · 2026-09-15
- Grouped Value Attention shrinks KV cache by reconstructing keys on demand — Vishesh Tripathi · 2026-09-15
- Amazon's MInTRL uses sparse off-policy interventions to boost on-policy RL — amazon · 2026-09-15
- Stateless LLM failover preserves ~0% context; ContinuityBench proxy hits 99.20% CPR — its_vayishu · 2026-09-15
- Phillip Isola highlights a non-mainstream AI route: RL from scratch via ultra-fast simulators — AjdDavison · 2026-09-15
- SSAD2026 talk covers autonomous driving 3D perception, from LiDAR self-supervision to multi-sensor distillation — abursuc · 2026-09-15