# Inside Hermes Agent Traces with NeMo Relay | Nemotron Labs

## Executive summary

The session details how agent traces, using Hermes Agent and NeMo Relay, transform opaque tool-use workflows into structured, inspectable telemetry. This is critical for enterprise adoption of agent fleets, enabling AI teams to perform robust AI evaluations, optimize models, and maintain compliance. NeMo Relay acts as an open-source observability layer that captures model calls, tool usage, and runtime state while supporting advanced governance features like PII redaction and data fencing for personal versus shared company IP.

## Key takeaways

- Agent Traces vs. System Logs: Unlike traditional system logs used for root cause analysis, agent telemetry captures the full lifecycle of an agent's turn, including every model call and tool execution. This data serves as valuable AI evaluation and training data for improving models and operational processes. (0:01:30)
- Operational Model Shift in Enterprise Agents: Moving from a personal agent (where the user is the operator) to a managed agent fleet introduces a split between the user and the operator. In large enterprises, the operational model further splits, requiring a dedicated 'AI team' responsible for tuning skills and models, necessitating advanced telemetry for measurement and troubleshooting. (0:02:15)
- NeMo Relay's Multi-Layer Observability: NeMo Relay provides four layers of agent observability: basic split/messaging health checks (L1), agent analytics (measuring token calls, LLM calls, turns without storing content), full trajectories (L4), and full audit/forensic storage (L3). This allows companies to tailor their data retention policies based on compliance needs. (0:03:10)
- Governance and Data Fencing: The system supports data fencing: personal agents can be configured to strip out all content and identifiable information, retaining only analytics and diagnostics, while shared agents (company IP) can store full content for auditing and knowledge capture. (0:03:45)

## Technical details

- NeMo Relay Functionality: NeMo Relay is an open-source NVIDIA library integrated into the agent's execution path. It performs three core functions: **Observe** (records every step, model call, and tool), **Govern** (enforces policies, such as preventing specific commands like `rm-rf`, and automatically redacting PII using regex), and **Optimize** (enables caching and model routing). (0:04:15)
- Agent Fleet Architecture: The 'Team Gateway' acts as a central router, directing incoming messages to the appropriate personal or shared agent within the fleet. This architecture is designed to manage complex, multi-user agent interactions. (0:03:55)
- Hermes Agent Capabilities: Hermes is highlighted for its self-learning capability; it improves and creates its own skills over time as it is used, making it a unique differentiator compared to platforms relying solely on plugins. (0:01:55)

## Practical implications

- Build engineers can instrument their own agent harnesses using the principles demonstrated by NeMo Relay to capture comprehensive, auditable telemetry.
- The system allows for granular control over data retention (data fencing), enabling the deployment of agents in highly regulated environments (e.g., separating personal employee data from core company IP).
- The observability layer facilitates rapid root cause analysis and troubleshooting of complex agent failures (e.g., deprecated models or failed skills) by providing detailed, traceable spans.
- The concept of 'profiles' within the agent allows a single agent instance to perform different roles or tasks with distinct capabilities, enhancing flexibility.

## Topics

Agentic Workflows, Observability, AI Governance, LLM Operations (LLMOps), Enterprise AI, Hermes Agent, NeMo Relay, LangSmith, Phoenix, OpenTelemetry

Source: https://www.youtube.com/watch?v=7HTRAsVX10E