How to regression test LLM agents when tool-call order varies every run
evaliqa · reddit · 2026-10-21
Strict sequence assertions on agent tool calls go red half the time for no real reason. The author's proposed approach:
- Compare sets, not order: assert the multiset of tool calls contains required ones; check order only where real dependencies exist (no refund confirmation before order lookup)
- Required/forbidden lists per scenario: must call lookuporder; must never call deleteaccount or issuerefund without prior verification — forbidden calls hard-fail on a single run
- Fixed N repeats scored as pass rate: e.g. N=10 at production temperature, seed pinned where supported
- Statistics over single runs: Fisher's exact test or confidence intervals to compare baseline vs candidate pass rates, failing builds only on statistically meaningful drops
Open questions: N=10 gives little power (a 90%→70% drop is hard to detect reliably); whether to run CI at temperature 0; how to tolerate harmless extra calls without masking real issues.
More from coding & agent
- Claude cut a 2.6-hour video export to 20 minutes — but only after being told to rebuild the server — byhewar · 2026-10-23
- 3 flight-booking rules my AI agent checks that I'd never do by hand — Efistoffeles · 2026-10-23
- OpenJet claims Qwen3.8-27B coding runs 1.9x faster than llama.cpp on one RTX 3090 — JeremyCMorgan · 2026-10-23
- Building an autonomous SEO agent in Hermes with three skills: programmatic pages, Keywords Everywhere, DataForSEO — EXM7777 · 2026-10-23
- Building a sports betting AI agent with onchain execution and human approval by default: architecture and lessons — Dry-Comfortable-2678 · 2026-10-23
- How do you audit AI agent decisions in regulated workflows? A field survey — Ok_Way_1993 · 2026-10-23