BATCH: Behavior Alignment for Cross-Embodiment Learning from Humans

Stanford University

Paper Code (coming soon)

BATCH overview: behavior alignment between human and robot data, used for co-training and human-to-robot style transfer in a cross-embodiment video action model — **BATCH** performs behavior alignment between diverse out-of-domain human data and limited in-domain robot teleop data, accounting for differences in execution speed and irrelevant human segments. The aligned pairs are leveraged through two objectives within a downstream cross-embodiment video action model: co-training and human-to-robot style transfer. The resulting policy transfers visual knowledge from human data to robot deployment in out-of-domain scenarios.

Why Behavior Alignment?

Human demonstrations provide a scalable source of manipulation data, but they are not simply additional data from a different visual domain. As the two videos below illustrate, the human completes the task much faster than the robot and includes an extra segment to reset the cup that has no counterpart in the robot trajectory. Prior co-training approaches typically sample human and robot trajectories independently, so paired clips may correspond to different stages of the task—for example, the human may be grasping the cup while the robot is placing it on the saucer. BATCH instead performs behavior alignment, identifying frame-level correspondences between equivalent stages of task execution across embodiments.

Human demonstration

Robot demonstration slower

Human

Robot

Method

BATCH has two parts. (A) Behavior alignment learns a shared temporal embedding from unaligned human and robot demonstrations and uses DTW to retrieve behavior-aligned segment pairs. (B) A cross-embodiment video action model trains on these pairs with joint co-training and human-to-robot style transfer objectives.

(A) Behavior Alignment

(B) Cross-Embodiment Video Action Model

Alignment Results

Given a segment of a robot video, BATCH retrieves the human demonstration of the same behavior from human scenes with different backgrounds and objects, and aligns it frame by frame with the robot segment. BATCH typically retrieves human segments that closely match the robot's behavior, while baselines often produce phase mismatches or include irrelevant actions.

Robot segments hover to select

BATCH-retrieved human video

Policy Results

We start from a policy pretrained only on in-domain robot demonstrations, then post-train it with just 5–8 human demonstrations per scenario, recorded in scenes the robot has never seen. We test the robot in these novel scenes on two real-world tasks: Object in Container, with new backgrounds and new target and distractor objects, and Cup on Saucer, with new objects and environments. BATCH achieves the strongest performance on both. Behavior alignment and the human-to-robot style-transfer objective together enable the most post-training transfer, and removing either one hurts. Pairing the same human data with robot trajectories at random instead of by behavior (BATCH w/o alignment) drops the score from 0.63 to 0.42 on Object in Container and the success rate from 83.3% to 50.0% on Cup on Saucer, and post-training on the same human data with plain co-training instead of style transfer (EgoVerse) also falls short of BATCH on both tasks.

1 · Pretrainrobot demonstrations only, in-domain scenes

→

2 · Post-train with BATCH+ 5–8 aligned human demonstrations per new scenario

→

3 · Deployrobot in scenes seen only in human data

BATCH rollouts (played at 4×). In the charts, BATCH w/o alignment randomly samples human and robot trajectories instead of using behavior-aligned pairs; EgoVerse co-trains on human and robot data without the cross-embodiment style-transfer objective; BATCH w/ OT replaces style transfer with an optimal-transport loss on observation tokens; Robot Only uses no human data.