7 engineering principles for reliable agents, distilled from hundreds of practitioners

kevinlu310 · reddit · 2026-11-05

Building on the author's earlier piece (DeepSeek's super-agent architecture, five principles — build agents like distributed systems, not chat apps), hundreds of engineer comments were distilled into seven engineering principles:

1. Independent supervision: liveness is not progress

A worker can be alive, consuming tokens, retrying the same failed step for an hour
Define progress at task-design time: expected outcomes, milestones, verifiable checkpoints
Measure persisted state, not activity — stalled state means possibly stuck regardless of logs
Give the watchdog enforcement authority (pause/terminate/restart); a watchdog without a kill switch is just logging
Supervision must be independent: an agent checking its own work isn't supervision; workers can make steady, measurable, useless progress
Practice chaos engineering: inject stuck workers, measure time-to-detection
K8s analogy: agents need readiness probes for task progress; one engineer saw the fastest model "finish" a lab task in 1.2 minutes without solving it — speed without progress is just expensive

2. A failure catalog of six production problems, including:

Crash After Action: worker completes an external action but crashes before recording it; fix with durable pre-action checkpoints, reconcile against the system of record, preserve unresolved business state
Zombie Workers: a supposedly killed worker returns and corrupts state (post truncates here)

The remaining principles and the full catalog are in the original article.

Original post →

7 engineering principles for reliable agents, distilled from hundreds of practitioners

More from coding & agent