NVIDIA Minerva AIOps Platform Connects AI Factory Telemetry to Maximize GPU Efficiency
Public- 26 Oct, 2026

AI factories represent some of the most complex operational environments ever built, where hundreds of thousands of GPUs, NVIDIA Spectrum-X Ethernet fabrics, NVLink scale-up networks, and distributed schedulers operate in continuous lockstep. In synchronous distributed training, the entire job advances at the pace of its slowest rank, meaning a single degraded GPU, interface, or switch path can waste useful time across many healthy resources. The failures that hurt yield are often subtle, such as a GPU running slightly hotter than its peers, an NVMe device losing performance under sustained load, or a switch generating pause-frame storms. These signals rarely cross static thresholds individually but collectively create stragglers, extended step times, and delayed checkpoints, leading to significant production losses.

Diagnosis is further complicated because evidence is split across different teams: network operators see interface counters, compute operators see node health, and ML teams see changing step times. This fragmentation turns triage into a chain of dashboard searches and cross-team handoffs, potentially turning a contained fault into days of lost production time. NVIDIA Minerva addresses these gaps by collecting telemetry across the entire stack and mapping infrastructure to active jobs in a dynamic knowledge graph. It applies purpose-built machine learning to correlate dependent symptoms into high-fidelity incidents. An AI agent-powered assistant and an MCP server connected to external agentic tools provide a conversational path through this live evidence, with the primary goal of preserving effective GPU hours and keeping model training moving smoothly.

Rule-based monitoring struggles with "gray failures" where components remain available but exhibit issues only under load or in comparison with peers. Absolute thresholds often miss these shifts, while aggressive thresholds generate noise during normal workload changes. Scale amplifies this problem, as one physical fault can produce interface errors, congestion indicators, and GPU utilization changes that appear as unrelated events on separate dashboards. Minerva ingests metrics, logs, and topology from GPUs, Spectrum-X and NVLink switches, SuperNICs, DPUs, storage, NCCL, and schedulers. Auto-discovery supplies relationships that raw telemetry lacks, creating a knowledge graph that maps infrastructure, scheduler, and workload elements. This graph updates dynamically as jobs start, stop, or nodes are replaced, supporting navigation from a factory-wide view down to an individual interface.

The topology graph transforms the investigation path by allowing operators to trace anomalies from an interface to its node, fabric path, and current job placement. Conversely, ML engineers can trace a job ID through GPU, host, NIC, and switch evidence. Minerva processes billions of metric and log data points per hour using supervised and unsupervised models. Supervised detection identifies known critical error patterns and associates them with remediation guidance, while unsupervised models learn normal patterns for hardware or logical components and compare them with relevant peers. Peer comparison is crucial because a measurement may look acceptable in isolation but anomalous in context, such as a GPU temperature within specification but significantly hotter than sibling GPUs under the same load. Multivariate models combine temperature, power, clocks, and utilization to detect degradation before outright failure.

Logs receive similar treatment, with NLP-based anomaly detection surfacing previously unseen patterns and supervised matching identifying known critical errors. Detections can feed scheduler workflows to drain affected nodes before new jobs land on them or at safe checkpoint intervals. Topology-aware correlation connects detection to operator action by leveraging the knowledge graph to determine which events share a dependency path. Downstream symptoms are grouped into a single incident rather than competing for attention, collapsing redundant events and related metric series into one signal with affected topology and workloads attached. NVIDIA testing shows significant reductions in notified events and tickets for supported scenarios, allowing operators to begin with a much smaller search space for root-cause analysis.

NVIDIA Minerva includes a native AI agent called Sherlock, grounded in live telemetry and the knowledge graph, which allows operators to query the system for root-cause analysis in human language. Sherlock can present supporting evidence and recommend remediation without requiring operators to know specific metrics or query languages, providing a route to domain knowledge that traditionally required specialist escalation. Additionally, Minerva's MCP server lets external agentic tools query the same operational context, enabling teams to incorporate live AI factory evidence into their own workflows. Mitigation actions can remain human-approved or be automated according to local policy and risk assessment. In a large AI data center, Minerva identified a single NVLink connection out of 400,000 interfaces losing bandwidth, allowing an SRE to drain affected nodes and restore full speed in minutes rather than hours.