NVIDIA AICR adds k8s-aibom to track AI workloads in Kubernetes GPU clusters

  • PublicPublic
  • 12 Nov, 2026
NVIDIA AICR adds k8s-aibom to track AI workloads in Kubernetes GPU clusters

Version-locked platform recipes describe how a GPU cluster should be built, but they do not show which AI workloads actually run on it. Teams deploying through NVIDIA Dynamo, Kubeflow or Slurm update workloads separately from the recipe. Mutable tags like :latest make this worse: two nodes can run different code under the same tag, and a later pull may fetch an image no security review ever covered. NVIDIA AI Cluster Runtime (AICR), an open source component of the NVIDIA DSX AI Factory Platform, now includes a qualified AI bill of materials component. It is based on k8s-aibom, a Google open source project that tracks AI workloads on Kubernetes.

Figure 1. NVIDIA DSX brings together the technologies needed to design, build and operate AI factories
Figure 1. NVIDIA DSX brings together the technologies needed to design, build and operate AI factories

The two tools work at different layers. AICR turns known-good GPU configurations into version-locked recipes and reproducible bundles. A recipe targets one combination of hardware, cloud, OS and intent, and records components, versions, values, constraints, deployment order and expected validation checks. Bundles include checksums, and aicr bundle --attest adds optional signing. k8s-aibom watches workload and pod APIs and emits a CycloneDX 1.6 AIBOM for each recognized AI workload. It supports native Dynamo graphs, NIMService, LeaderWorkerSet, DisaggregatedSet and CronJob, and child workloads roll up into their owner's AIBOM. Qualification means AICR pins the upstream source, chart, CRDs and a digest-pinned controller image as one tested set. AICR also verifies provenance against Google's build identity, reviews RBAC and security defaults, and tests install, upgrade, health and uninstall.

In the recipes h100-gke-cos-inference and h100-gke-cos-inference-dynamo enable k8s-aibom by default. Other GKE recipes can enable it with --runtime-inventory enabled, and turns it off --runtime-inventory disabled. The examples use AICR v1.2.0, whose catalog pins k8s-aibom 1.6.2, and run on GKE with Kubernetes 1.32 or later and H100 nodes on Container-Optimized OS with GKE-managed drivers. The recipe is generated by aicr recipe --service gke --accelerator h100 --os cos --intent inference, and the resolved values are returned by aicr query with --selector components.k8s-aibom.version. aicr bundle builds the bundle with system and accelerated node selectors. The Helm deployer is the default; helmfile, flux and argocd are also supported. After aicr verify and deploy.sh the controller runs in the k8s-aibom-system namespace. Helm bundles apply the main chart's CRDs, while Flux and Argo CD update them through their controllers. In Helmfile, Terraform and Fleet, CRDs must be updated manually if a transition requires it; the expected storage version is v1beta1.

The Team aicr snapshot, followed by aicr validate --phase deployment --phase conformance, runs the k8s-aibom health check. It requires an available controller, an AIBOMControllerConfig with Ready=True and the expected CRD storage version. A matching storage version does not prove the full schema is current, so during upgrades also check for Degraded=True with reason SchemaPredatesController. Inventory is enabled per namespace with the label aibom.k8saibom.dev/enabled=true. The label limits where AIBOM records are created, but the informers still cache workload and pod specs cluster-wide, including arguments and inline environment variables. No sink and no secret access are configured by default. In the demo, a vLLM Deployment serves Qwen/Qwen2.5-0.5B-Instruct from the vllm/vllm-openai:latest image. Its AIBOM is classified with category inference and runtime vllm. Records are built from the spec, so an AIBOM can show Ready=True before the pod finishes pulling its image: it does not prove scheduling or execution.

The BOM is stored inline on the resource along with a SHA-256 hash of the canonical bytes: it can be extracted with jq and base64, and the hash checked with shasum. Its components list the application (vllm), the container image (vllm/vllm-openai) and the machine-learning model (Qwen/Qwen2.5-0.5B-Instruct). A conventional SBOM identifies the image, while an AIBOM also identifies the model and the serving runtime. Each attribute carries a confidence tier (declared, inferred, unresolved or verified) and an evidence locator pointing to the source the value was taken from. The k8s-aibom developers measured the controller overhead of version 1.6.1 on GKE with Dynamo and AICR's configuration. The test used CPU-only nodes, and the workload pods were left pending.

Several gaps are intentional. Runtime observations are not signed yet, although the canonical bytes and hash make signing practical. k8s-aibom can verify Sigstore signatures on model claims, enforce signer identity and check Rekor inclusion, but AICR disables this by default. Linking AICR bundle attestations to runtime inventory is future work. So is comparing recipe intent with observed workloads, which requires shared artifact identities, freshness rules and coverage mapping. AICR is available at NVIDIA/aicr and k8s-aibom at GoogleCloudPlatform/k8s-aibom; both are open source.