Scene reconstruction for robot simulation, now in closed beta

Robots rarely deal with one object at a time. They open the drawer under the counter, move between the table and the shelf, and reach past whatever is in the way. To train and test that, you need the whole room.

Scene generation builds that room. Start from a description, a few photos, a walkthrough video, a 360 capture, a floor plan or a scan, and you get back an editable scene you can load straight into your simulator. The objects come from the same pipeline as asset generation, our open SimReady asset library, and your existing workspace assets, meaning that with use over time, the cost per scene generation will go down.

You can ask for a brand-new space or a close copy of a real one. If your footage only covers half a room, the director can fill in the rest.

Every scene is built by a scene director, an agent that plans the work, checks its own output and keeps going until it’s satisfied with the results.

The scene director works in drafts. The first drafts block out the scene, setting the position and rough shape of walls, openings, and large furniture. Each later draft adds smaller objects, materials, lights, and other high-fidelity details.

You can preview each draft as it’s made and give feedback in the agent chat. The director works your feedback into the next draft.

Flying around a 3D scene every time you check on it gets old quickly. Instead, the scene director sets up a few saved views, like an overview, interior shots and close-ups, and every draft comes with a rendered preview from each one.

Each saved view stays at the same camera position across every draft, which makes it easy to compare draft three with draft seven. For reconstructions, each draft is also shown next to your original frames, so you can see what still doesn’t match.

The output is USD, allowing for Omniverse-based sensors along with ovrtx-enabled rendering. Similarly, a scene isn’t useful if it stutters in simulation, so we optimize every one.

Scenes can be generated from a variety of sources, including egocentric videos, Gaussian splats, and large-scale walkthroughs. Here are some ways to use them:

• Train mobile manipulators in homes and workplaces where the drawers and cabinets actually open.
• Test a policy in a copy of the room your robot will work in, built from a walkthrough video.
• Start from one generated scene and vary the layout and objects for more robust training.
• Produce perception training data with consistent layouts and object identities across views.
• Build the environment around a new task instead of modeling it by hand.

• It’s a closed beta, so quality, speed and the review tools will continue changing.
• Indoor scenes are the current focus. Outdoor scenes work too, but we haven’t tuned or optimized them as much yet.
• Without a size reference, like printed markers, known dimensions or a floor plan, scale is estimated.
• Anything the director fills in, like the far side of a room your video never shows, is plausible rather than measured. The scene records which parts were observed and which were generated.
• Large scenes with many objects can take several hours to generate and use more tokens.