The Robot Data Problem: Why Training Robots Is So Hard

Language models had an enormous collection of text to learn from. A robot needs information that ordinary text and images rarely contain: what its cameras saw, what its motors did, and what happened after contact. The physical equivalent would be a labeled record of every grasp, push, and pull a person makes, with the forces and outcomes attached. That record does not exist and cannot be scraped. Three properties make physical experience uniquely expensive to collect:

Recording each attempt takes equipment, operator time, and repeated resets.

One research direction that may narrow the gap is gaining momentum in 2026: world models. NVIDIA's research builds robot policies on a video model that predicts how a scene will evolve, so that more kinds of interaction footage can serve as training signals.

• It is embodied. A demonstration is only directly useful to a robot with a similar body. Watching a person yank a stuck drawer does not transfer cleanly to a two-finger gripper.
• It is consequential. A language model can predict a wrong word harmlessly. A robot exploring by trial and error snaps handles, over-pulls drawers off their rails, and wears out hardware. Mistakes cost real money.
• It is slow. The real world runs at one second per second. A robot practicing physically gathers a bounded number of attempts per day, and the rare failure cases (the drawer that sticks, the handle that is slippery) show up the rarest of all.

The task stays the same: open a drawer, including styles the robot was never trained on. Each source offers different information.

Simulation is the only one of the three where the drawer itself is something you build. Here is what that object has to be: not a picture of a drawer unit but a working one, with every drawer a separate body on its own joint.

The cost drivers differ by source, and the numbers only mean something with their task and collection conditions attached.

• Teleoperation: paid human time per robot-hour. It costs operator time, equipment, resets, and recording review. Collection speed depends on the task and setup. For scale, the public DROID dataset holds 350 hours of robot interaction, which took 50 data collectors at 13 institutions about 12 months to gather.
• Human video: cheap to gather, expensive to use. It may be easier to obtain, but filtering, labeling, and adapting it for robot control take work. Acquisition cost alone says little about its eventual usefulness.
• Simulation: cheap per attempt, but not free. It can reduce the cost of repeated attempts once a usable scene exists. Building and validating assets, integrating the controller, and running the compute remain part of the bill, so simulation shifts cost toward building and maintaining the world rather than removing it.

NVIDIA describes the training mix for its GR00T N1 humanoid model as a pyramid, with data quantity decreasing and embodiment specificity increasing from base to peak:

The proportions vary by project. NVIDIA reports generating over 750,000 synthetic trajectories in 11 hours in its GR00T N1 pipeline, which it equates to about nine months of human demonstration data. It also reports a 40% performance gain from combining synthetic and real data, compared with real data alone. These results are specific to its model and experiments; the generation time excludes the work of preparing the pipeline.

• Base (largest): internet-scale web data and human video, providing broad visual and linguistic information, but no robot motor signals.
• Middle: synthetic data generated in simulation, subject to differences between simulated and real behavior.
• Peak (smallest): real robot data collected by teleoperation, the most faithful and the most costly per example.

A failure does not identify its cause on its own. A consistently misplaced grasp may come from camera calibration, while a drawer that jams in simulation may have an incorrect joint. Before collecting more data, run conditional checks: calibration (camera and frame transforms), control (command tracking and timing), and asset (collision openings, joint limits, mass and inertia). The table and diagnostic below describe possibilities to investigate, not a diagnosis.

The sources overlap. More visual variety may help one failure while leaving another untouched. NVIDIA's RoboLab benchmark makes a similar split, tagging each task by visual, procedural, and relational competency rather than reporting one blended score.

Simulation supports repeated training attempts and evaluation across varied scenarios. NVIDIA's RoboLab work calls it "the natural place to run large-scale robot evaluations."

That usefulness depends partly on the objects. The drawer unit you drove above only works as training data because each drawer is a real joint with a real limit; a handle likewise needs collision geometry that supports grasping. Preparing those assets takes work before training can begin, and it is where most of the sim-to-real risk hides. A viewer can show authored articulation and travel, but it does not guarantee physics, contact, friction, or controller validity.

Palatial helps turn objects into assets with these properties for robot simulation. Our guide to objects with moving parts explains what that involves.

Simulation fidelity is never perfect, and video does not directly measure contact forces. Real demonstrations also cover only the conditions recorded. No source covers every task or deployment condition.

Reported gains are model-and-task specific, and a success rate is only as trustworthy as the number of trials behind it. NVIDIA's RoboLab work gives a concrete example: the same observed 90% success rate means very different things at 70 rollouts and at 1,030.

Estimating one policy's reliability is also a different question from comparing two policies, which needs more trials than most published benchmarks run. Held-out scenarios and real-world validation remain necessary before deployment.