[ sources ]

Where the action labels come from

Robotics teams don't treat data collection as a choice between methods anymore — they combine them. Teleoperation brings fidelity, simulation brings scale, and human video brings diversity. What really separates these sources, and decides what each one is useful for, is whether the data arrives with action labels or you have to recover them after the fact.

Robot-in-the-loop

Teleoperation & kinesthetic teaching

An operator drives a physical robot through a task using VR controllers, leader-follower arms, exoskeletons, or by physically guiding the arm. The robot's own actuators record the demonstration. It's imitation learning, not trial and error, and pooled cross-embodiment teleop data now underpins most of the leading generalist policies.

Action labels: Full, native
Embodiment gap: None
Throughput: ~5–50 episodes per operator-hour
Real cost: Operator time and robot hardware

Robot-free demonstration

Handheld & wearable interfaces

A human demonstrates the task directly, using instrumented equipment instead of a robot: handheld grippers with fisheye cameras and encoders, motion-capture gloves, or sensorized rigs. It's faster than teleoperation because there's no robot in the loop, but it still needs someone deliberately sitting down to collect.

Action labels: Partial (end-effector pose, grasp state)
Embodiment gap: Small
Throughput: Higher; no robot required
Real cost: Rig cost, and active collection effort

Passive capture

Egocentric human video

First-person video of people doing real work, captured with head-mounted cameras or AR glasses in unscripted environments. It scales further than any other source and captures whole-body coordination and tool use that no control interface can reproduce. The catch: video shows a hand closing on an object, but not the force applied or the joint angles of an equivalent arm. The actions have to be inferred, and the inference is noisy.

Action labels: None; must be inferred
Embodiment gap: Large
Throughput: Highest
Real cost: Curation, annotation, de-identification

Synthetic

Simulation

Demonstrations generated in physics engines with perfect state information and infinite replay. Essential for locomotion, rare events, and anything unsafe to capture for real. Policies trained purely in simulation still degrade on contact-rich tasks, which is why sim usually gets paired with domain randomization, photorealistic rendering, digital twins, and a round of real-world fine-tuning.

Action labels: Perfect, free
Embodiment gap: Sim-to-real gap
Throughput: Effectively unlimited
Real cost: Scene authoring and fidelity engineering

In deployment

The data flywheel

Robots already operating in the field record every episode and feed it back into training. It's the most valuable data source there is, and the only one that compounds — but only if you already have robots in the field.

Action labels: Full, native
Embodiment gap: None
Throughput: Compounding
Real cost: Requires an existing fleet

Source	Action labels	Embodiment gap	Throughput	Real cost center
Teleoperation	Full, native	None	Low	Operator time, hardware
Handheld & wearable	Partial	Small	Medium	Rig cost, active collection
Egocentric video	None (inferred)	Large	Very high	Curation, de-identification
Simulation	Perfect	Sim-to-real	Unlimited	Scene authoring
Deployment flywheel	Full, native	None	Compounding	Requires deployed fleet