Why traces cannot tell whether an agent actually used its retrieved context
PersonalityWhich1780 · reddit · 2026-07-29
This long post argues that execution traces alone cannot tell you whether an agent actually used the documents it retrieved.
The author explains the difference between:
- retrieval succeeded, but the answer did not use it
- retrieval was weak, but the model still relied on it
Because control-flow traces only show that the retriever ran and passed documents downstream, they miss whether the generator consumed that evidence. The post proposes a few ways to estimate “used,” including leave-one-out regeneration, lexical overlap, and possibly NLI entailment.
The author also shares a practical lesson from their own knowledge-graph retriever over GitHub events: highly connected hub nodes were being retrieved constantly but rarely used, and naive node-level suppression silently discarded 47% of PRs. They fixed this by suppressing hubs at the node-plus-relation level instead of per node alone.
The tool, GraphSight, is MIT licensed, runs locally, and is pip-installable.
More from coding & agent
- Five open-source tools promise 60%–95% fewer tokens for Claude and Codex agents — CodeByPoonam · 2026-07-29
- Codex Micro should be voice-first, with Git and PR controls replacing approve/reject — rudrank · 2026-07-29
- Agent teams should use deterministic checks first, then LLM judges, then humans — yoobinray · 2026-07-29
- Matt Pocock turns a wayfinding map into a full Cmd+K palette spec — mattpocockuk · 2026-07-29
- GitHub spec turns a diagram playground into a Cmd+K command palette — mattpocockuk · 2026-07-29
- Codex can now handle trims, subtitles, masking and title cards in basic video edits — gabrielchua · 2026-07-29