Audit Finds Serious Bugs in 206 of 600 AutomationBench Verifiers

Parsewave audited all 600 public tasks of the widely cited Zapier AutomationBench and found serious bugs in 206 verifiers, raising doubts about the benchmark's scoring reliability and model rankings based on it.

2026-10-07 ~ 2026-10-07 · 2 related posts

• Parsewave audit finds 206 serious verifier flaws in 600 public AutomationBench tasks — rohanpaul_ai · 2026-10-07
• Audit finds 206 buggy graders in Zapier's AutomationBench; fixed scores flip 27.9% of runs — rohanpaul_ai · 2026-10-07