Les développeurs de NVIDIA utilisent des agents IA de pointe pour automatiser les flux de travail de simulation Omniverse

Les développeurs combinent de plus en plus les modèles IA de pointe avec les bibliothèques NVIDIA Omniverse pour transformer les concepts de simulation en applications fonctionnelles. Cette intégration permet aux équipes d'assembler des actifs numériques, de connecter des moteurs physiques et de configurer des pipelines de rendu avec un effort manuel considérablement réduit. En guidant les agents IA par des instructions en langage naturel, les ingénieurs peuvent se concentrer sur les objectifs de conception de haut niveau, tandis que les agents gèrent l'implémentation technique complexe. Les applications résultantes prennent en charge l'exploration de scénarios, l'investigation des pannes et l'amélioration itérative de la conception dans diverses industries. Ce changement de flux de travail permet un prototypage plus rapide et des environnements de test plus robustes pour les systèmes complexes.

Frank DeLise, responsable produit Omniverse chez NVIDIA, a utilisé l'agent Astra pour créer un simulateur interactif pour l'automatisation d'entrepôts. Il a dirigé Astra pour connecter des bibliothèques Omniverse spécifiques, notamment ovphysx pour la physique, ovstage pour les mises à jour de scène, ovrtx pour le rendu et ovui pour l'interface utilisateur. En utilisant des actifs SimReady, l'agent a généré le code d'animation et d'application nécessaires pour afficher un robot humanoïde en vue de première et de troisième personne. Ce projet démontre comment l'IA peut automatiser l'assemblage de systèmes de contrôle basés sur la physique, permettant aux développeurs d'évaluer le comportement des tâches dans un environnement réaliste avant le déploiement de robots physiques. Le processus souligne l'efficacité de l'utilisation de l'IA pour combler l'écart entre les actifs statiques et les simulations dynamiques et interactives.

Doyub Kim, manager of the simulation technology team, used Astra to create Zero to Alpamayo, a reusable simulation environment based on Market Street in San Francisco. Kim guided the agent to map out the workflow and connect asset creation, traffic simulation, and driving models in stages. The resulting prototype serves as a testing ground for comparing different driving models and tracing how scene or sensor changes affect downstream behavior. Additionally, a separate experiment using Cosmos3-Nano varied weather and lighting conditions in recorded simulation videos. This allowed Kim to analyze the driving model’s responses to identical scenarios under different environmental factors, providing critical data for refining autonomous vehicle algorithms.

Ashley Reid, working on RTX sensor validation, directed Astra and Claude Fable 5 agents to compare simulated sensor outputs with recorded real-world data. The agents created two digital twins from scratch and improved two existing ones over a three-day iterative workflow. They measured differences between ovrtx camera and raw LiDAR outputs, creating or modifying OpenUSD scenes to address missing objects, geometry, and materials. Acceptance criteria were based on specific camera and LiDAR metrics, ensuring that the simulated sensors closely matched their physical counterparts. This methodology provides developers with a systematic way to use measured discrepancies to guide scene creation, ensuring high-fidelity validation for autonomous systems and robotics applications.

Tae Kim, leading NVIDIA Omniverse engineering, used sports videos and natural language instructions to guide Astra in building Robo Olympics. This experimental project tests simulated Unitree G1 humanoids performing various sports movements. Under Kim’s direction, Astra built controllers and refined them through physics trials using the Newton Physics Engine and the open-source NVIDIA Warp framework. The ovrtx library rendered the scenes and virtual camera images, providing visual feedback on the robot’s performance. In one specific experiment, the robot successfully cleared a single hurdle in 64 out of 100 simulation trials. These trials provided valuable feedback for improving the robot’s timing and control mechanisms, demonstrating the utility of AI in optimizing physical constraints and movement planning.

Jens Jebens, a senior product manager for OpenUSD, directed Astra to model a car suspension in PTC Onshape and configure it in NVIDIA Isaac Sim. The agent measured available space and designed a wrench that the robot could use to reach the suspension’s bolts. Jebens reported the successful removal of a suspension component in the simulation, connecting design and tooling decisions directly to disassembly results. This workflow offers a starting point for robot policy training, ensuring that tools are viable before physical manufacturing. Nic Johns, an engineering director, also used Astra to assemble NASA assets into an OpenUSD International Space Station model with telemetry. He built the application with a single prompt, using Blender for asset preparation and Omniverse libraries for rendering and streaming, bringing 3D models and operational data into the browser.