How to Generate Articulated, Simulation-Ready Assets (and Why It's Hard)
In robotics simulation, "articulated" means the model contains an internal kinematic structure: two or more rigid bodies connected by joints. Everyday examples dominate robot-training environments: cabinets and drawers, doors, laptops, appliances with lids, scissors, bottles with screw caps, valves.
If a robot's task involves opening, closing, twisting, or folding something, the training environment needs objects that behave correctly when handled.
What counts as an articulated asset?
A complete, simulation-ready articulated asset specifies, at minimum:
- Part decomposition: each independently moving component as its own body, and everything that doesn't move fused into its parent. Over-splitting creates unstable, jittery physics; under-splitting produces the classic failure, a dresser whose drawers are welded shut.
- A parent-child joint hierarchy. This is the 101 of joint authoring: bodies form a kinematic tree, and each joint connects a child body to its parent. Move the cabinet and its drawers come along; slide a drawer and the cabinet stays put.
- Joint definitions: type (revolute for hinges, prismatic for drawers and sliders, fixed for rigid attachments, and coupled types where supported), axis placement, and travel limits.
- Physics per body: mass, center of mass, inertia, and collision geometry for each part, not just the whole.
- Joint dynamics: friction and damping so a drawer doesn't fly open at a touch or refuse to move.
- Simulator-native format: OpenUSD physics APIs for NVIDIA Isaac Sim, MJCF body trees for MuJoCo, URDF for ROS-adjacent tooling.
Why is this hard for AI generation?
1. Part structure is invisible in the source
A photo of a closed drawer looks identical to a photo of a drawer-shaped carving. Geometry alone cannot tell you what moves. Single-image and text-to-3D diffusion models generate one continuous surface, which is why outputs from general-purpose generators are usually fused meshes with painted-on seams. Recovering structure requires semantic understanding of the object category (drawers slide, doors hinge), explicit user intent ("the cap unscrews"), or source data that carries structure, like CAD assemblies with real part hierarchies.
2. Joint inference is a physical reasoning problem
Even with parts correctly separated, the pipeline must decide type, axis, and limits for every connection, and assemble them into the right tree. Get a hinge axis a few millimeters wrong and the door intersects its frame; pick revolute instead of prismatic and the drawer swings like a gate. Research systems tackle this with learned joint predictors; production pipelines combine category priors, geometric analysis (detecting hinge cylinders, slide rails, thread regions), and agent-based reasoning about the object's intended use.
3. "Passes generation" is not "works in simulation"
A jointed model can render beautifully and still fail the moment a policy interacts with it: parts interpenetrate at joint extremes, unrealistic inertia makes a door oscillate forever, or a reinforcement-learning agent discovers it can push a drawer through the cabinet back. That last failure class is really a collision-generation problem: the drawer and cabinet each need collision geometry tight enough that contact is enforced everywhere a policy can probe, and readiness tags (such as SimReady-style profiles) are only trustworthy when collision quality is part of what the tag certifies. The only reliable acceptance test is actuating every joint through its full range in the target simulator, plus adversarial interaction testing for RL use.
What are the main approaches today?
Four approaches dominate, each with a different trade-off between control, coverage, and cost.
| Approach | How it works | Strengths / limits |
|---|---|---|
| Manual authoring | Artists build parts and rig joints by hand in DCC tools, then export USD/MJCF | Highest control; slow and expensive; the baseline everything else is trying to beat |
| Procedural / parametric | Template families with parameterized dimensions and joints | Reliable structure by construction; limited to templated categories; strong for domain randomization |
| Research pipelines | End-to-end models predicting parts and joints from images or meshes | Promising coverage; physical validity varies; rarely production-packaged with collision, physics, and validation |
| Agentic generation | An AI agent interprets intent, decomposes parts, selects joints from geometric and semantic evidence, authors physics, and routes through validation | Handles open-vocabulary objects; this is the approach Palatial takes, with simulator-in-the-loop checks rather than blind trust |
How do you specify what you need?
Whatever pipeline you use, ambiguity is the enemy. Three rules that consistently improve results:
- State what moves, explicitly. "A cabinet with two doors that open on side hinges and one drawer that slides out" beats "a cabinet." Structure that isn't stated must be guessed.
- State the task. "For robot manipulation training: the robot will grasp the handle and pull the drawer open" tells the pipeline the drawer needs a working slide, realistic friction, and a graspable handle.
- State scale. Kinematics amplify scale errors: a hinge authored on a 3-meter cabinet behaves absurdly. Real-world dimensions anchor mass and inertia too.
Format notes: USD vs MJCF
- OpenUSD (Isaac Sim): the moving structure is expressed through physics APIs on the prim hierarchy: an articulation root on the assembly, rigid-body and mass APIs per part, joint prims connecting parent and child bodies, with drives and limits as attributes. One correctness rule worth memorizing: single rigid objects must not carry an articulation root. Isaac Sim's joint vocabulary also includes coupled types like the rack-and-pinion joint (linking prismatic and revolute motion); Genesis offers similar coupled joints.
- MJCF (MuJoCo): the body tree is the hierarchy, nested body elements with joint children, inertial and geom elements per body. MuJoCo's solver conventions differ from PhysX, so a model validated in one simulator still needs its own pass in the other; cross-simulator parity is a real workstream, not a file conversion.
What does "good" look like?
- Every intended joint exists, with the correct type, a physically sensible axis and limits, and the right parent-child relationship in the tree.
- Full range-of-motion actuation in the target simulator without interpenetration or instability.
- Parts at rest stay at rest (no drift or jitter); moved parts settle plausibly.
- Collision geometry per part, tight enough for manipulation, hollow where the real object is hollow, and certified by whatever readiness tag the asset carries.
- For RL: survives adversarial interaction, forces beyond normal use don't glitch parts through each other.
- Provenance: which joints were inferred vs specified, so downstream users know what to verify.
Limitations and open problems
- Screw threads: true helical joints are poorly supported across simulators. Production pipelines typically approximate a screw cap with a constrained revolute joint plus friction, and where the simulator supports coupled joints the twist-advance behavior can be modeled more faithfully. Fine for most training tasks; still short of torque-accurate assembly simulation.
- Ambiguity is irreducible: no pipeline can know whether a particular cabinet's left door is fixed decoration without being told. Systems that ask one good clarifying question outperform ones that guess.
- Category long-tail: common categories (cabinets, laptops) work far better than rare mechanisms (folding ladders, umbrella linkages) in every current approach, including ours.
- Validation coverage: actuating joints proves the kinematics, not task fidelity; whether a specific policy transfers to the real object remains a separate sim-to-real question.
Frequently asked questions
- Can text-to-3D models produce assets with working joints?
- Not directly. They produce single meshes; the moving structure must be added by a downstream pipeline that segments parts, infers the joint tree, and authors physics. Production use additionally requires collision geometry, mass properties, simulator-native export, and validation.
- What input produces the best results?
- CAD assemblies with real part hierarchies are the strongest source, structure comes for free and should be preserved rather than regenerated. Next best: images or text plus an explicit statement of what moves and what the robot will do with it.
- How long does generation take vs manual authoring?
- Manual rigging typically runs hours to days per asset. Automated pipelines are dramatically faster where they succeed, with the caveat that hard cases may need a correction pass.
- How do I check an asset before training on it?
- Actuate every joint through its full range in your actual simulator, verify the parent-child tree behaves correctly, run rest-stability and interaction tests, and inspect per-part collision geometry.