Agent regression testing benchmark: embeddings caught only 7/19 real answer changes
redhatpanda07 · reddit · 2026-09-23
The author built a 109-pair fixture with ground truth by construction (19 real changes, 90 rewordings) to benchmark detectors for agent prompt-edit regression. Exact match: 19/19 caught but 84/90 false alarms; embeddings under 0.75 caught only 7/19; a markers→embed→LLM-judge stack got 19/19 with 33 false alarms; a 4-way Choice classifier hit 19/19 with 12 flags at $0.0028 and 0.43s. Key lesson: false alarms, not recall, are the metric — every false alarm costs a human re-read.
More from coding & agent
- Building a Custom Agent Harness with Pi and Jev, With Interactive Playground — dair_ai · 2026-09-23
- Why AI-Edited UI Keeps Breaking, and the Visual Feedback Loop Fix — nikola_mr64990 · 2026-09-23
- Long AI sessions degrade suddenly because context accumulates garbage, not because models got worse — ClickOk5811 · 2026-09-23
- Same model, same prompt, worse API extraction: apps do invisible pre-processing you're skipping — TangeloOk9486 · 2026-09-23
- Asked AI to Review a Known Concurrency Bug, It Came Back Clean — cto_junior · 2026-09-23
- One line that changes everything: 'If you have any questions, ask me' when prompting AI — thisiskp_ · 2026-09-23