Agentic apps need different MLOps: caching, model routing and evals for low latency

kmeanskaran · x · 2026-09-18

Drawing on personal production experience, the author explains why MLOps for agentic projects differs fundamentally from classic ML.

Key points:

The hardest part is cutting latency while keeping accuracy — only achievable through trial and error.
Clean logging, observability and dedicated dev/UAT environments are mandatory; otherwise you're "praying for next tokens."
Chunking strategy matters like feature engineering did in classic ML.
Four latency levers: high-quality data ingestion, caching to avoid repeated tool calls, better agent system design, and model routing to keep small models on simple queries.
Latency issues mostly come from searching large chunks and repeated multi-tool calls; accuracy work should focus on agent harness and evals.
CI/CD is easy; the real difficulty is Terraform and environment separation. A follow-up article on running agents in production is promised.

More from coding & agent