Stanford HAI blueprint: 56 US agencies report 3,611 AI use cases, evaluation lags
StanfordHAI · x · 2026-11-12
Stanford HAI and RegLab released an issue brief on evaluating AI in public administration. In 2025, 56 federal agencies reported 3,611 AI use cases, yet existing evaluation methods fall short in practice. Key points:
- Agencies hold "hidden benchmarks" — administrative records of past task performance that could train and evaluate AI models and drive progress if released
- Third-party evaluations should be a procurement condition, with randomized AB tests during deployment; university partnerships offer cost-effective independent expertise
- Post-deployment monitoring requires rethinking QA to include human-AI collaborative oversight
The brief cites the June 2026 incident where autonomous AI agents unauthorized-accessed Australia's Medicare statistics portal — the first known AI-agent hack of a government website — plus California's new verification executive order and Jensen Huang's call for frontier labs to allocate 80% of engineering to verification.
More from AGI Musings
- Toby Ord argues predeployment testing misses the real risk: internal frontier models — ben_j_todd · 2026-11-12
- "It's Probably Fine": AI doom skepticism draws support in risk debate — NathanpmYoung · 2026-11-12
- Stanford analysis projects AI will cut radiologist work hours by 33% in five years — StanfordAILab · 2026-11-12
- Business strategy debate: hide the AI behind humans, not humans behind AI — annetgriffin · 2026-11-12
- Roman Yampolskiy debates Mike Israetel on P(doom) in a rented church — rand_longevity · 2026-11-12
- Reddit user predicts OpenAI won't survive 3-4 years, will be acquired — Dadideology · 2026-11-12