NVIDIA and Nous Research detail agent tracing with NeMo Relay across 108-run eval
NVIDIAAI · x · 2026-10-01
NVIDIA's tech blog, built with Nous Research/Teknium, shows how NVIDIA NeMo Relay captures execution traces for the Hermes Agent and turns them into controlled harness evaluations.
- What NeMo Relay does: a common observability layer for agent runs; Hermes natively emits ATOF event streams, ATIF step-by-step trajectories, and OpenTelemetry spans with OpenInference labels viewable in Arize Phoenix.
- Two demos: a terminal-tool task where Hermes runs a script in an isolated Docker container with exact output verification (VALUE=42, zero tool errors), and a multi-tool research task that reads a travel record, searches the web, verifies the COLT 2026 conference on its official site, and writes a report.
- Key result: a Hermes ToolPerf case study compared baseline vs. fixed revisions across 108 runs — Qwen Coder 30B recovered more tasks but increased calls, data volume, and latency. Fixed isn't the same as more efficient.
- A companion repo ships runnable examples, verifiers, and Phoenix setup.
Related event: NVIDIA and Nous Research Detail NeMo Relay for Agent Tracing(2 posts)→
More from coding & agent
- Firecrawl's Alexandria hits #6 most popular ChatGPT plugin with 120+ data providers — devdigest · 2026-10-01
- Cloudflare launches agent-themed batch: pay-per-use gateway, AutoRouter, 6x faster containers — threepointone · 2026-10-01
- Blogger warns the cheap vibe-coding era is almost over — and the dots explain why — michalmalewicz · 2026-10-01
- Pocket FM's Memory System Matches Claude Code's 86.7% Accuracy at 1/21 the Cost — bigaiguy · 2026-10-01
- Ben Goertzel: AI agents are co-authoring formally verified software, not just patching bugs — bengoertzel · 2026-10-01
- Pedro Domingos: There are no software engineers anymore, we're all agent wranglers — pmddomingos · 2026-10-01