Testing Non-Deterministic LLM Agent Pipelines with Property-Based Assertions
Repulsive_Sugar_5252 · reddit · 2026-09-18
The author builds hospilot, where an LLM dynamically decides which agents form a pipeline and in what order based on the goal text — the core feature that makes it useful for tasks no one predefined.
The problem: near-identical prompts return a different agent graph on each run. Both are usually valid, just different — fine when a human is watching, but it breaks traditional testing's determinism assumption (same input, same output) when you want to assert 'given this goal, the pipeline should look like X.'
Current workaround: property-based assertions instead of exact-match — does the graph include the agents required for this goal type, respect the constraints passed in, and avoid out-of-scope stuff? Weaker than exact matching, but it tests something real rather than 'did the output match last Tuesday.'
The author asks if anyone has found something better: testing LLM decisions that are allowed to vary but not allowed to be wrong feels like its own discipline, and most conventional testing advice doesn't map onto it.
More from coding & agent
- DeepSeek V4.1-Flash vs V4-Pro Benchmarked: 3x Cheaper but Slower and Weaker Output — Arindam_1729 · 2026-09-18
- The Agent-Coding Failure Nobody Talks About: Right Code, Wrong Idea — gandazgul · 2026-09-18
- Open-source MCP connector lets you drive the JSON-only Jev model via Claude — purealgo · 2026-09-18
- Junior backend dev asks: is learning RAG, vector DBs and agents worth it yet? — Effective_Emphasis21 · 2026-09-18
- WebMCP draws developer buzz as a way to bring MCP into the browser — nickbaumann_ · 2026-09-18
- Demo: Claude with built-in Cowork and Artifacts builds a multiplayer game from one prompt — felixrieseberg · 2026-09-18