Amazon Paper Finds Scoring Flaws in Rule-Based AI Benchmarks

An Amazon team audited nine rule-based AI benchmarks and found errors in answer keys and scoring code that misjudged correct model responses, releasing a revised benchmark set called BAR.

2026-11-09 ~ 2026-11-09 · 2 related posts

• Amazon paper finds rule-following AI benchmarks mark correct answers wrong; one scorer fix lifts LegalBench scores by 80+ points — rohanpaul_ai · 2026-11-09
• Follow the Policy, Fail the Benchmark: Amazon audit finds nine policy benchmarks flawed, releases BAR revision set — rohanpaul_ai · 2026-11-09