NVIDIA Deploys AI Agents to Automate AI Factory Operations and Reduce SRE Workload

  • PublicPublic
  • 05 Nov, 2026
NVIDIA Deploys AI Agents to Automate AI Factory Operations and Reduce SRE Workload

As GPU clusters scale, site reliability engineers (SREs) face the challenge of correlating performance drops across compute, network, and hardware layers. NVIDIA developed the AI Factory Operations Agent to streamline internal DGX SuperPOD operations by reducing manual effort. The system uses specialized agents that detect failures, correlate root causes, and provide recommendations via a chat interface. Engineers can interact with these agents using natural language, switch between investigations, and track actions for auditability, replacing the need to check multiple dashboards manually.

Figure 1. AI Factory Operations Agent architecture: a NemoClaw agent coordinating domain-specific agents across the GPU cluster infrastructure
Figure 1. AI Factory Operations Agent architecture: a NemoClaw agent coordinating domain-specific agents across the GPU cluster infrastructure

The developer example is built on the NVIDIA NemoClaw blueprint, featuring a user-facing agent runtime that routes queries to domain-specific agents. Connectivity is established through OpenClaw plugins and Model Context Protocol servers, which expose focused operational tools. The framework utilizes NVIDIA OpenShell and supports fully on-premises deployment to ensure data sovereignty. Inference relies on locally served NVIDIA Nemotron open models, running across DGX H100, B200, and GB300 production clusters. This modular design allows for the addition of new agents and infrastructure integrations without modifying existing components.

Deployment occurs as a Kubernetes control-plane application, with Helm managing the UI, OpenClaw gateway, MCP servers, and inference endpoints. For shell execution, OpenShell creates per-session Kubernetes sandboxes to isolate commands from the main gateway pod, ensuring the runtime does not modify production workloads. Security is enforced through layered access controls. The server-side run_kubectl tool uses pinned binaries and read-only credentials, while Kubernetes RBAC defines inspectable resources. This setup allows agents to verify permissions, such as confirming they can get pods but not secrets or create new ones.

The internal deployment utilizes a suite of domain-specific agents. The Job Recovery Agent analyzes training failures across scheduler and hardware layers. The Hardware Agent triages device-level faults like PCIe errors and thermal anomalies. The Observability Agent queries Prometheus and Grafana Loki to generate dynamic dashboards. The Research Agent searches preloaded documentation, while the Kubernetes Agent provides read-only awareness of cluster state. The Cluster Management Agent interfaces with Base Command Manager APIs. Cross-cutting tools include web search and code execution, and the framework supports custom MCP servers for specific operational needs.

NVIDIA recommends a dedicated GPU-accelerated control plane isolated from the production cluster to handle agent orchestration without consuming training resources. In multi-tenant environments, OpenShell enforces boundaries by running agents in isolated sandboxes with kernel-level restrictions, ensuring that tenant SREs only access their specific namespaces and nodes. The framework is available as experimental open source software on GitHub, though APIs and configurations may change between releases. NVIDIA customers can contact their account managers to request access to the proprietary cluster operations agents developed for internal use.