Standard Bank automates genAI risk assessment on AWS with LLM-as-a-judge across 300+ controls
AWS ML Blog · rss · 2026-11-21
AWS ML Blog details how Standard Bank Group, with AWS Professional Services, automated governance approvals for generative AI use cases.
Problem: Each new genAI app needed manual review and testing by the risk committee, making governance the slowest pipeline step.
Architecture: Runs entirely in the bank's own AWS account — Well-Architected Tool for intake (custom lens with 40 risk questions), Step Functions orchestration, S3 evidence storage, and a foundation model on Amazon Bedrock acting as an LLM-as-a-judge (swappable via config, prompts never shared with model providers).
Four-step pipeline:
- A pre-assessment agent on Fargate scans the code repo and docs, pre-populating the lens with answers and evidence links for human review;
- The pipeline checks system docs, policy docs, source code and system prompts, plus live testing: built-in jailbreak and off-topic probes, repeated/reworded prompts with embedding-based semantic consistency checks;
- LLM-as-a-judge returns yes/no per check; 300+ checks map to international AI management/risk/security standards and internal policies across eight responsible-AI dimensions;
- Each risk gets a likelihood score (share of missing evidence) plus adjustable severity, yielding Very low to Critical ratings traceable to specific standard clauses for regulators.
The risk committee keeps final approval but now decides on consistently collected evidence.
More from coding & agent
- Matt Pocock: ADRs are essential for coding agents — don't remove a fence before you know why it's there — mattpocockuk · 2026-11-21
- Matt Pocock on ADRs: keep them short, only for surprising hard-to-reverse decisions — mattpocockuk · 2026-11-21
- SwiftTerm maintainer: ~80% of PRs now involve AI, in distinct waves — film_girl · 2026-11-21
- Open-source MCP server gives coding harnesses a shared long-term memory with your personal agent — Empty_Pizza9768 · 2026-11-21
- Cloudflare Browser Run /snapshot adds PDF, links, scrape, and JSON in a single page load — dinasaur_404 · 2026-11-21
- Chat's core flaw for agent work: scrolling back for old artifacts constantly breaks focus — max__drake · 2026-11-21