DOCKET benchmark shows LLM policy feedback tilts toward government: 68% overlap vs 26% for unions
stanfordnlp · x · 2026-11-23
Diyi Yang and coauthors release DOCKET (arXiv:2611.17914), a benchmark for evaluating pluralistic AI-assisted policy feedback in U.S. rulemaking.
- Dataset: 42.3M public comments on 22,400 regulations across 44 agencies (2016-2026), linked to specific policy items to track public attention and regulator responsiveness.
- DOCKETBench: only 18-30% of LLM-generated feedback aligns with real human comments; LLMs critique broadly without grounding, citations, or lived-experience arguments.
- Key skew: LLM feedback overlaps with up to 68% of government claims but as little as 26% of labor union claims, and persona modeling does not fix the bias.
More from Safety
- Fake official account spotted, prominent blogger warns links are unsafe to click — dotey · 2026-11-23
- Why cybercriminals skip AI attacks: too noisy, costly tokenomics, old TTPs still work — jcran · 2026-11-23
- Researchers Push Back on Perplexity Safety Bars: Data Mixture Bias Undermines Cross-Lab Comparisons — eliebakouch · 2026-11-23
- Vercel CEO Rauch: 'free' ZDR stealth models look like industrial-scale data exfiltration — AccBalanced · 2026-11-23
- Agents Sabotage Peer Shutdown in 38.3% of Runs Across 17 Models, No Task Needed — rohanpaul_ai · 2026-11-23
- Australia's Spy Agency Now Treats Data Center Ownership as Foreign Interference Risk — rohanpaul_ai · 2026-11-23