AWS shows how to route coding agent requests by cost per solved task with NeMo Switchyard on Bedrock

AWS ML Blog · rss · 2026-11-14

AWS ML Blog published a full walkthrough for routing coding agent requests (Claude Code, OpenAI Codex CLI, etc.) across models on Amazon Bedrock using open-source NVIDIA NeMo Switchyard with a LiteLLM gateway.

Key ideas:

Coding agents are billed per task, and one task may span dozens of model calls; token price alone doesn't reflect total cost, which also depends on turns, history replay, and retries/failed tasks.
Routing should therefore be judged by cost per solved task: total spend divided by runs whose tests pass.

Solution details:

Switchyard offers Stage routing (reads tool activity in request history to pick between a capable model and a cheaper efficient one) and weighted Random routing for traffic splits/A-B tests; neither needs a judge model call.
Deploy via AWS CDK as a LiteLLM+Switchyard container on ECS Fargate in private subnets, fronted by WAF and an ALB path allowlist, calling Bedrock Converse API through a VPC interface endpoint.
Sample benchmark runners compare a routed group against single-model baselines (Claude Opus 5.5 and Nemotron 3 Super 120B) so you can evaluate routing on your own workloads.
Nvidia documents switchyard-server 0.3.0 as experimental/demo; config and APIs may change, non-production use only.

Original post →

AWS shows how to route coding agent requests by cost per solved task with NeMo Switchyard on Bedrock

More from coding & agent