DOCKET Benchmark: Only 18-30% of LLM Policy Comments Align With Real Human Feedback
stanfordnlp · x · 2026-11-21
Stanford-led researchers released DOCKET, the most comprehensive dataset of the U.S. Federal Notice & Comment rulemaking process, and DOCKETBench, the first systematic eval of LLMs representing human policy perspectives.
- The dataset spans 42.3M public comments on 22,400 regulations across 44 agencies (2016-2026), linking comments to specific policy items to track regulator responsiveness.
- Only 18-30% of LLM-generated feedback aligns with real human comments: LLMs get the tone right but lack the specific grounding, citations, and lived-experience arguments humans bring.
- Some models skew systematically toward government perspectives (68% overlap) while agreeing with labor groups on as little as 26% of claims—a skew even persona modeling doesn't fix.
More from Safety
- Epic Used Claude Security Model to Find MyChart Audit-Log Blind Spots, Paused Dev 6 Weeks — pswider · 2026-11-21
- Puppet control-plane CVE (CVE-2026-115731) exercise: patched systems show unauthorized changes 3 months later — Dave_Maynor · 2026-11-21
- Hands-on: discovering local AI agents on Windows via Defender for Endpoint — davemccollough · 2026-11-21
- Signal Restricts Screenshots to Block AI Agents from Reading Messages — ZeroStateReflex · 2026-11-21
- Debate on X: most frontier-lab safety roles net negative, but Perez and Nanda are exceptions — austinc3301 · 2026-11-21
- Threat actor claims leak of 815 million Indian Aadhaar records on darkweb — kingslyj · 2026-11-21