How Airbnb runs coding agents on Bedrock: cost per task drives model picks and effort ceilings

AWS ML Blog · rss · 2026-11-19

On the AWS ML Blog, Airbnb details how it operates its internal coding-agent tool AirChat (thousands of weekly users) on Amazon Bedrock. With 8 new models in 5 weeks and 2/3 of requests flowing to them by mid-October, Airbnb runs evals on its own 7 task suites (built from real code changes) scoring both quality and cost per task, then sets model defaults and reasoning-effort ceilings enforced at its LLM gateway. Post-rollout, CloudWatch Bedrock metrics track latency (TTFT, OTPS), throttling (EstimatedTPMQuotaUsage for quota alarms), and four token cost types including cache pricing. A real data point: raising one model's effort from low to high gained 0.8 score points but cost 2.5x per task. Includes a practical checklist: streaming-only TTFT, invocation logging per region, cost allocation tags activated up front, dated price tables.

Original post →

More from coding & agent

coding & agent channel →