Using Grounded Theory for Agent Behavior Analysis at Scale
Zhuoran Lu, Yangyang Yu, Zhuoyan Li, Yibo Meng, Nan Jiang, Chengxi Zang, Jie Gao, Ziang Xiao
EMNLP 2026)
cs.CL, cs.AI
2026-08-31
AutoTraceGT runs grounded theory over 7,500+ traces. Codebooks cover 73-91% of human failure modes, add missed categories, and as features beat or complement few-shot predictors.
SWE-bench tells you whether an agent finished the job. It does not tell you what the agent did for a few hundred steps. Human reading does not scale. Frozen failure taxonomies miss behaviors that show up on new tasks. Length and action counts scale, and they are process-blind.
Grounded theory, used in the social sciences for six decades, grows categories from data instead of testing a pre-written list. It stops at theoretical saturation, when new samples stop adding structure. Nobody had turned that loop into an auditable, stoppable pipeline for agent traces. Earlier LLM coding systems skipped stages or stopped after a fixed number of rounds.
AutoTraceGT uses four roles. OpenCode tags one trajectory with 2-to-5-word conceptual codes, a step span and a verbatim quote, chunking long traces and carrying a memo. AxialCode groups a batch into categories and typed relations, tagged success, failure or both. Manage reconciles new categories into a running codebook with add, merge, split or flag, merge-first. Saturation is two consecutive rounds with new-category rate below ε=0.2. TheoreticalCode then reads only the saturated codebook and writes a core category plus a narrative. No new local evidence enters at that stage.
Two corpora. Tau-Bench, Go-Browse and SWE-Agent contribute about 2,000 outcome-labeled traces each. ALFWorld, GAIA and WebShop come with expert failure types. Backbones are GPT-4.1-mini, GPT-5, GPT-5-mini and GPT-OSS-120B. Default batch size is 30 traces, 50 messages per open-coding chunk.
Adds taper, merges and confirms rise, and codebook cosine approaches the terminal state, which is the signature saturation is supposed to have. Three disjoint reruns of the same dataset-model cell have median codebook cosine 0.929. Across 594 cross-cell pairs the median is 0.791 and the 95th percentile is 0.901. Holding the dataset fixed and swapping backbones still beats a permutation null that shuffles dataset labels (p<0.001).
| Dataset | Human / machine cats | Recall | Precision | Trace match |
| ALFWorld | 15 / 20 | 75.0 | 60.0 | 82.4 |
| GAIA | 11 / 17 | 73.7 | 63.2 | 58.0 |
| WebShop | 10 / 13 | 90.9 | 88.9 | 87.9 |
Recall sits above precision because the codebook grows classes the human list lacks: producing no admissible action at all, announcing a plan change that never happens, oscillating between paging and resets. The theoretical narrative lands on an action stream that has stopped integrating environmental feedback and keeps replaying the original error, the same cascade prior experts described, now at the level of observable behavior rather than internal modules. A one-pass codebook on the same backbone loses recall and match on all three sets.
As GLM features, "click a visible control" in Go-Browse has β=-2.59 (success); click then close without checking has β=+3.28 (failure). In SWE-Agent a sweeping workaround has β=+4.10 (failure); building a minimal reproducer after reconnaissance has β=-1.82 (success). For failure prediction the AutoTraceGT codebook does not always beat a few-shot codebook on its own; the union often does. GPT-5 complementary features reach MCC 0.499 and AUC 0.828 on Go-Browse.
The coverage judge is GPT-5. On 100 pairs, two graduate annotators agree at κ=0.74; agreement with the judge is 0.66 and 0.76. The judge is conservative.
If the question is what agents actually do, not pass@k, this is an inductive path that runs to thousands of traces with a quote behind every code. It fits offline review, training-data mining and process-level rewards. It does not fit per-step online control. Treat the codebook as hypotheses for human review, not as ground truth about agent populations.
Every coding stage is an LLM. Cross-backbone stability rules out one vendor's private bias and does not rule out blind spots shared by frontier models. Coverage numbers are conditional on a GPT-5 judge. The pipeline makes many calls and runs to saturation, so it is an offline tool. The study is English, single-agent, success/fail. Algorithmic saturation is not the methodological object. Only three corpora have expert failure labels; the other three offer a weaker external anchor for the theoretical story.