Using AI Agents to Prepare 3D Scenes for Simulation
Public- 17 Sep, 2026

Robotics simulation engineers routinely hit the same bottleneck, and it has little to do with policy, models, or training loops. The hard part often begins before any learning happens: the robot still lacks a simulation-ready world to train in. Getting a 3D scene from a Blender file into a state usable by a simulator is tedious, repetitive, and easy to get wrong, and it frequently falls outside the scope of the simulation engineer's job. That prep work is exactly where agentic systems can help, provided they carry the right tools.
The workflow described here connects reasoning, tool execution, and validation into a single repeatable process. Codex, powered by OpenAI GPT-6 Astra, acts as the orchestrator that sets the objective, interprets results, and guides iteration. Specialized subagents built with the Hermes agent harness and deployed through NVIDIA NemoClaw call NVIDIA Omniverse Libraries to inspect the scene, author simulation metadata, configure physics, and render visual preflight views. The stack starts from a concrete objective: the input is a Blender scene, the goal is a robotics-ready world, the output is a USD-based simulation-ready scene, the destination is Isaac Sim or Isaac Lab, and validation combines visual preflight with SimReady checks.
The first subagent connects to Blender through a Model Context Protocol (MCP) server and inventories the scene rather than guessing from screenshots. It can call tools to inspect objects, collections, transforms, materials, cameras, lights, and scene metadata, returning a structured inventory that becomes shared context for the rest of the agents. A sample result counts 142 objects and 37 materials while flagging missing semantic labels, collision meshes, camera sensors, and physics materials. Astra uses that inventory along with the robotics objective to identify missing information, plan the next tasks, and delegate them with clear acceptance criteria.


The inventory then flows into specialized agents that act on the scene. A USD authoring agent preserves hierarchy, transforms, materials, labels, physics metadata, and sensor definitions, using USD's layered, nondestructive composition so agents can enrich the world without flattening the original artwork. A semantic-labeling agent turns anonymous meshes into task-aware objects such as shelves, bins, floors, obstacles, grabbable items, and robot targets, inferring labels from names, hierarchy, shape, and context while flagging uncertainty. In the demo, 118 prims were tagged and 9 labels left for human review. A material agent converts visual appearance into simulation-useful metadata, identifying metal shelving, cardboard boxes, plastic bins, concrete floors, rubber wheels, or glass panels rather than chasing prettier look-dev.

A sensor subagent authors camera and lidar devices early so that placement, field of view, range, polling rate, occlusion, and target visibility can be validated before the scene reaches the simulator. The ovphysx subagent then makes the world physically aware, adding collision shapes, mass, friction, and rigid-body behavior, and turns its findings into a repair plan. It generates simple collision meshes for static props, marks floors and shelves as fixed colliders, assigns rigid-body properties to grabbable objects, and flags anything that depends on task intent. Safe, mechanical fixes run automatically, while ambiguous cases, such as an uncertain semantic label or physical behavior, are escalated to a human with context and a proposed next step. The result is a physics-readiness report that downstream tools can consume.