NVIDIA Developers Use Frontier AI Agents to Automate Omniverse Simulation Workflows

  • PublicPublic
  • 09 Oct, 2026
NVIDIA Developers Use Frontier AI Agents to Automate Omniverse Simulation Workflows

Developers are increasingly combining frontier AI models with NVIDIA Omniverse libraries to transform simulation concepts into functional applications. This integration allows teams to assemble digital assets, connect physics engines, and configure rendering pipelines with significantly reduced manual effort. By directing AI agents through natural language instructions, engineers can focus on high-level design goals while the agents handle the intricate technical implementation. The resulting applications support scenario exploration, failure investigation, and iterative design improvements across various industries. This shift in workflow enables faster prototyping and more robust testing environments for complex systems.

Frank DeLise, an Omniverse product manager at NVIDIA, used the Astra agent to create an interactive simulator for warehouse automation. He directed Astra to connect specific Omniverse libraries, including ovphysx for physics, ovstage for scene updates, ovrtx for rendering, and ovui for the user interface. Using SimReady assets, the agent generated the necessary animation and application code to bring a humanoid robot into a first- and third-person view. This project demonstrates how AI can automate the assembly of physics-based control systems, allowing developers to evaluate task behavior in a realistic environment before deploying physical robots. The process highlights the efficiency of using AI to bridge the gap between static assets and dynamic, interactive simulations.

Doyub Kim, a manager on the simulation technology team, employed Astra to build Zero to Alpamayo, a reusable simulation environment based on San Francisco's Market Street. Kim guided the agent to map out the workflow and connect asset creation, traffic simulation, and driving models in stages. The resulting prototype serves as a testing ground for comparing different driving models and tracing how scene or sensor changes affect downstream behavior. Additionally, a separate experiment using Cosmos3-Nano varied weather and lighting conditions in recorded simulation videos. This allowed Kim to analyze the driving model's responses to identical scenarios under different environmental factors, providing critical data for refining autonomous vehicle algorithms.

Ashley Reid, working on RTX sensor validation, directed Astra and Claude Fable 5 agents to compare simulated sensor outputs with recorded real-world data. The agents created two digital twins from scratch and improved two existing ones over a three-day iterative workflow. They measured differences between ovrtx camera and raw LiDAR outputs, creating or modifying OpenUSD scenes to address missing objects, geometry, and materials. Acceptance criteria were based on specific camera and LiDAR metrics, ensuring that the simulated sensors closely matched their physical counterparts. This methodology provides developers with a systematic way to use measured discrepancies to guide scene creation, ensuring high-fidelity validation for autonomous systems and robotics applications.

Tae Kim, leading NVIDIA Omniverse engineering, used sports videos and natural language instructions to guide Astra in building Robo Olympics. This experimental project tests simulated Unitree G1 humanoids performing various sports movements. Under Kim's direction, Astra built controllers and refined them through physics trials using the Newton Physics Engine and the open-source NVIDIA Warp framework. The ovrtx library rendered the scenes and virtual camera images, providing visual feedback on the robot's performance. In one specific experiment, the robot successfully cleared a single hurdle in 64 out of 100 simulation trials. These trials provided valuable feedback for improving the robot's timing and control mechanisms, demonstrating the utility of AI in optimizing physical constraints and movement planning.

Jens Jebens, a senior product manager for OpenUSD, directed Astra to model a car suspension in PTC Onshape and configure it in NVIDIA Isaac Sim. The agent measured available space and designed a wrench that the robot could use to reach the suspension's bolts. Jebens reported the successful removal of a suspension component in the simulation, connecting design and tooling decisions directly to disassembly results. This workflow offers a starting point for robot policy training, ensuring that tools are viable before physical manufacturing. Nic Johns, an engineering director, also used Astra to assemble NASA assets into an OpenUSD International Space Station model with telemetry. He built the application with a single prompt, using Blender for asset preparation and Omniverse libraries for rendering and streaming, bringing 3D models and operational data into the browser.