Postman's Postmortem on Agent Mode for 40M Devs: Context, Not Capability, Is the Bottleneck

AWS ML Blog · rss · 2026-10-09

Postman and AWS detail the architecture behind Agent Mode, the AI-native interface serving 40 million developers. Core lesson: demo vs. production are different engineering problems — the hardest parts weren't model quality or prompts, but making a mature product legible to an agent.

Controlling tool sprawl: Early highly-atomic tools made long workflows slow (every step round-tripped through the model), and testing showed tool-selection errors rose once the visible toolset exceeded 40 tools. The current architecture uses a vector DB to narrow 170+ tools to 15 per request for a context-isolated sub-agent, and decouples tools from UI state (e.g., sending requests in background without open tabs).

Schema-based reads: For structured data like the API Catalog, instead of one tool per question, the agent gets ClickHouse table schemas and generates complex queries with joins and WHERE clauses — shifting work from tool-building to data modeling once.

Context was the real bottleneck: Missing or wrong context caused more failures than missing capabilities. The design splits automatically gathered, minified background context from user-selected deep context routed through per-entity handlers.

Production controls include user approval for state-changing actions, task-scoped tools, PII redaction via Bedrock Guardrails, cross-region inference, and multi-tier prompt caching.

Original post →

Postman's Postmortem on Agent Mode for 40M Devs: Context, Not Capability, Is the Bottleneck

More from coding & agent