How to Build Enterprise-Grade Agents

hugobowne · x · 2026-07-17

This post discusses "how to actually build true enterprise-grade agents." The author interviewed a staff AI engineer at Maven Clinic, concluding that: you should strip decision-making power from the model and hand it over to application-layer rules and code.

Core Architectural Philosophy

Before the main agent sees a request, hard rules handle risks like prompt injection, off-topic queries, and self-harm.
The main agent's only job is to route requests to one of 4 specialized subsystems.
Each subsystem exposes only the tools it needs, keeping the entire system limited to about 15–20 tools.
The application layer directly injects known states (like user IDs) rather than letting the model generate them.

Evaluation & Launch Strategy

Programmatically checkable errors are handled by code to determine tool calls and failure reasons.
LLM-as-a-judge is used only for questions with "no definitive answer," such as clinical accuracy and diagnostic boundaries.
Humans then calibrate whether these judges align with real-world conversations.

Real-World Product Trade-offs

Some low-cost errors are acceptable. For instance, if the model claims "it was made by Google," a simple string check can handle it.
However, if an error could disrupt a medical process worth tens of thousands of dollars, it cannot ship, even if the overall capability looks impressive.

The overarching theme: enterprise agent reliability doesn't come from making the model "smarter," but from continuously narrowing the scope of what the model is allowed to decide.

Original post →

How to Build Enterprise-Grade Agents

More from coding & agent