A production-grade repo blueprint for LLM apps, from API layers to evals

techNmak · x · 2026-10-08

techNmak shares what he calls one of the most useful repository blueprints for production AI applications, organizing everything around the LLM into clear responsibilities:

api/: request handling, typed schemas, auth, rate limits
application/: workflows, business rules, budgets, output validation
providers/: model integrations, deadlines, bounded retries, cancellation
security/: identity, tenant isolation, access control, tool permissions
prompts/: templates, input contracts, model config, version references
observability/: tracing, latency, errors, usage and cost
evals/: versioned datasets, quality checks, held-out acceptance tests, sampled production evals
tests/ and deploy/: unit/integration/security/failure-path tests; builds, health checks, safe rollouts

RAG, agents, caching, tool execution, persistence and background workers are organized as separate, use-case-dependent capabilities, so a retrieval-heavy product and an agentic app both fit without forcing one architecture.

Key argument: model calls time out, users request unauthorized data, and prompt/model changes can degrade answer quality without breaking a single unit test — the app itself must handle these, with enough tracing, tests and evals. The repo structure won't make it work by itself, but gives each responsibility a clear home. Start small; add capabilities when the product needs them.

Original post →

A production-grade repo blueprint for LLM apps, from API layers to evals

More from coding & agent