A test pyramid for agents: five layers from tool unit tests to human review

ClickOk5811 · reddit · 2026-11-08

A Reddit post lays out a layered testing framework for LLM agents, arguing that not every part of an agent is non-deterministic and cheap checks should sit at the base.

Layer 1: tool unit tests — tools are ordinary code; test inputs, outputs, errors, timeouts. Many 'dumb agent' bugs are actually truncated tool results.
Layer 2: tool contract tests — verify the schema/descriptions the model sees and that invalid args yield helpful errors.
Layer 3: scripted-trajectory tests with mocked tools — assert on behavior, not wording: expected tool calls, forbidden actions blocked, step budgets respected, errors retried instead of hallucinated results. Run each case multiple times and track pass rates.
Layer 4: outcome evals on realistic tasks — run full tasks in a sandbox and check end states (records, labels, summaries), including hard cases like missing info and conflicting instructions; LLM graders are fine if validated.
Layer 5: human review of samples — read real runs including successes to discover new failure types, then push them down to cheaper layers.

Key principles: reproduce upper-layer failures with lower-layer tests, never assert on exact model text, and track pass rates over time. Limitations: all layers can pass on unseen inputs, and Layer 4 depends on sandbox fidelity.

Original post →

A test pyramid for agents: five layers from tool unit tests to human review

More from coding & agent