NVIDIA and Nous Research detail agent tracing with NeMo Relay across 108-run eval
NVIDIAAI · x · 2026-10-01
NVIDIA's tech blog, built with Nous Research/Teknium, shows how NVIDIA NeMo Relay captures execution traces for the Hermes Agent and turns them into controlled harness evaluations.
- What NeMo Relay does: a common observability layer for agent runs; Hermes natively emits ATOF event streams, ATIF step-by-step trajectories, and OpenTelemetry spans with OpenInference labels viewable in Arize Phoenix.
- Two demos: a terminal-tool task where Hermes runs a script in an isolated Docker container with exact output verification (VALUE=42, zero tool errors), and a multi-tool research task that reads a travel record, searches the web, verifies the COLT 2026 conference on its official site, and writes a report.
- Key result: a Hermes ToolPerf case study compared baseline vs. fixed revisions across 108 runs — Qwen Coder 30B recovered more tasks but increased calls, data volume, and latency. Fixed isn't the same as more efficient.
- A companion repo ships runnable examples, verifiers, and Phoenix setup.
Related event: NVIDIA and Nous Research Detail NeMo Relay for Agent Tracing(2 posts)→
More from coding & agent
- Generic API-catalog MCP server vs one tool per job: a context-bloat experiment — MandeepKaur9407 · 2026-11-16
- Berkeley paper: LLMs answer with stale values even when the update is in context — ombre-ai · 2026-11-16
- skills-gate: one gate skill fronts 148 agent skills at just 389 tokens — Distinct-Pie2389 · 2026-11-16
- Open-weight Karpo-Decision-31B tops Decision Index at 73.9, one forward pass per answer — CodeByPoonam · 2026-11-16
- shadcn boosts @uisdk: one-shot UI rebuild that's instantly E2E-testable by AI — shadcn · 2026-11-16
- AI agent deletes 43,599 lines of OpenClaw test code in one PR — vincent_koc · 2026-11-16