Follow the Policy, Fail the Benchmark: Amazon audit finds nine policy benchmarks flawed, releases BAR revision set
rohanpaul_ai · x · 2026-11-09
arXiv paper 2611.07042 (28 pages) by Yanru Xiao and Yifei Cai audits nine public policy-based decision benchmarks and finds unsupported targets, reference errors that survive replay or propagate as solved steps, and evaluators that reject acceptable answers or credit incomplete decisions.
Rescoring unchanged model responses quantifies the impact: fixing the scorer raises acceptance by over 80 percentage points for each of three models on the 96 LegalBench SARA numeric cases; restoring a required zero floor in RuleArena Tax changes 27 of 300 references and narrows an observed model gap; a controlled federal-exemption task shows errors in supplied intermediate answers can affect final decisions.
The paper releases BAR (Benchmark Audit and Revision), a versioned collection of audit evidence and revisions, with code and revised data open-sourced.
Related event: Amazon Paper Finds Scoring Flaws in Rule-Based AI Benchmarks(2 posts)→
More from Models
- Artificial Analysis launches Decision Index v2 with 45% private data and calibration-based scoring — ArtificialAnlys · 2026-11-09
- GLM-4.5 Hit #1 on Hugging Face Within 48 Hours of MIT-Licensed Release — ccerrato147 · 2026-11-09
- GLM-4's Closed API Model Lasted Just Four Months Before DeepSeek V2 Price Cuts — ccerrato147 · 2026-11-09
- Zhipu earned $110M but spent twice that on R&D — can open-weight models pay the bills? — ccerrato147 · 2026-11-09
- Has any LLM release ever been decidedly worse than its predecessor? — adonis_singh · 2026-11-09
- CEOs say Google's Argon stands out for writing, calling Google the AI dark horse — danshipper · 2026-11-09