BeyondRetarget: Learning Executable Humanoid Motions
Directly from Monocular Video

Tianyu Xiong^1,* Yi Lu^1,* Jinrui Wang¹ Ziqi Liang¹ Dandan Lei³ Xiaoyang Zhou⁴ Xiao-xiao Long² Qiu Shen^1,† Xun Cao¹

¹School of Electronic Science and Engineering, Nanjing University, Nanjing, China

²School of Intelligence Science and Technology, Nanjing University, Suzhou, China

³Jiangsu Mobile Information System Integration Co., Ltd., Nanjing, China

⁴China Mobile Zijin (Jiangsu) Innovation Research Institute Co., Ltd., Nanjing, China

^*Equal contribution · ^†Corresponding author

Beyond the conventional “human reconstruction → robot retargeting” pipeline.

End-to-end Multi-robot Contact-aware Streaming-ready

Code Hugging Face Live Demo

TL;DR BeyondRetarget maps monocular RGB video directly to executable humanoid motions by learning robot-oriented implicit representations from visual observations, without an explicit human-motion intermediate.

1 / 6

00:00 / 00:00

Multi-Robot Basketball Motion Capture

Interactive Online Demo

Turn Your Video into Multi-Robot Motion

Live on ZeroGPU

Upload a monocular video of up to 10 seconds. BeyondRetarget will generate motion packages and a synchronized visualization for eight humanoid robots using Hugging Face community ZeroGPU resources.

Notice: All videos and the Interactive Online Demo on this project page use the base version of our method. This version meets the real-time requirements of teleoperation, but does not yet support moving-camera scenarios or large-scale global human trajectories. We plan to open-source a more general and higher-performance version later with support for freely moving cameras, stronger global trajectory capture, and improved generalization across diverse visual conditions, including indoor and outdoor environments.

Multi-robot motion packages Eight-robot visualization Open demo in a new tab

Abstract

Learning executable motions from human videos offers a scalable solution for humanoid robots to acquire demonstration motions. However, existing pipelines typically first construct an explicit human motion representation and then convert it into robot motions via motion retargeting. Although such methods can effectively leverage large volumes of existing human data for training, the substantial differences between humans and humanoid robots in locomotion mechanisms and joint degree-of-freedom configurations make motions generated by this human-representation-centric approach difficult to execute on robots. Furthermore, errors introduced during human motion estimation inevitably propagate to the retargeting stage and cannot be eliminated via joint optimization. We propose BeyondRetarget, an end-to-end framework that directly maps monocular RGB videos to robot motions. Discarding the explicit human representation, this framework learns robot-oriented implicit representations directly from visual observations, enabling the model to capture cross-morphology motion structures. To generate motions more suitable for robot execution, we further design a contact-aware motion optimization mechanism to improve temporal consistency and physical plausibility. Experiments show that BeyondRetarget significantly improves the accuracy and robustness of generated robot motions, while achieving higher execution success rates and lower latency in both simulation environments and real humanoid robots.

End-to-End Estimation

Monocular RGB video maps directly to robot motion without an explicit human-motion intermediate.

Robot-Oriented Features

Implicit visual features capture motion structures across human and robot morphologies.

Contact-Aware Optimization

Contact-guided motion optimization improves temporal consistency and physical plausibility.

Executable Motions

Simulation and real-robot tests show greater accuracy, robustness, execution success, and lower latency.

Method

Conventional two-stage methods suffer from three fundamental limitations: an information bottleneck, because the retargeting stage cannot revisit the original visual evidence; error accumulation, because human-motion estimation and retargeting errors compound while the final robot-motion error cannot be propagated back for end-to-end optimization; and a robustness–adaptability trade-off. Optimization-based retargeting can support multiple robots, but its non-convex objective is prone to local optima and motion collapse. Learning-based retargeting can improve robustness, but adapting to a new robot requires a complete, high-quality robot-specific motion dataset and retraining the entire network from scratch.

BeyondRetarget replaces the disconnected two-stage pipeline with a shared motion encoder, robot-specific decoders, and a lightweight contact-aware refinement module. This unified design avoids the intermediate information bottleneck and cross-stage error accumulation while enabling efficient adaptation across robots, addressing the core limitations of two-stage methods.

Conventional two-stage video-to-robot pipeline versus BeyondRetarget's shared motion space and robot-specific decoders — Conventional two-stage retargeting VS End-to-end multi-robot motion estimation.

Step 01

Visual feature extraction

A frozen person detector and HMR2.0 visual backbone provide frame-level human-centric image features.

Step 02

Shared temporal encoding

Bidirectional GRU encoding, RoPE self-attention, and cross-window attention model local and long-range motion context.

Step 03

Robot-specific decoding

Residual output heads predict root trajectory, continuous root rotation, and robot-specific joint configurations.

Step 04

Contact-aware refinement

Temporal filtering, root correction, and constrained leg IK improve physical consistency while retaining motion intent.

BeyondRetarget architecture: visual encoder, temporal context aggregation, robot motion decoder, contact predictor, and refinement — Video features become a shared robot-oriented motion representation, which is decoded and refined into executable motion.

Main Results

G1 benchmark

End-to-end estimation VS two-stage baselines

Visual input unless marked GT · red / yellow = best / second best

Method	RAMPJPE ↓	RTE ↓	Jitter ↓	Accel ↓	Failure Rate (>65 mm) ↓	Failure Rate (>100 mm) ↓	Foot Slide ↓	Simulation SR ↑	Sim RAMPJPE ↓
Ours	26.36	80.74	1.41	1.51	0.00	0.00	1.71	95.86	35.04
GVHMR → NMR	39.19	701.13	1.63	1.56	6.21	2.22	2.28	93.92	43.32
GVHMR → GMR	39.46	637.33	7.94	2.51	3.43	0.28	3.30	86.46	41.77
WHAM → NMR	43.15	849.36	4.87	3.07	9.36	4.17	6.64	94.48	46.56
WHAM → GMR	45.88	217.76	63.26	14.57	10.47	6.30	13.30	81.49	42.89
GT → NMR	33.26	837.62	1.71	1.16	3.99	1.39	1.30	95.58	40.32
GT → GMR	32.02	36.01	10.65	2.27	0.00	0.00	0.82	95.86	35.93

Hands-on-waist and arm-swing motion comparison. All other two-stage methods suffer motion collapse.

Hands-behind-the-back motion comparison. Even with accurate SMPL estimation, the other two-stage methods still fail to place both hands behind the back.

Large arm circle motion comparison. All other two-stage methods suffer motion collapse.

Hand raise and squat motion comparison. During the squat, the two-stage methods place the hands in front of the head and fail to bend naturally at the waist.

Multi-Robot Evaluation

One shared motion feature, test on eight humanoid robots

RAMPJPE (mm) · lower is better

Method	H1	T1	Tienkung	G1	R1	GR1‑T1	GR2‑V3	Atlas
Ours	51.315	51.246	67.991	26.362	27.809	43.778	49.059	72.047
GVHMR → GMR	77.787	70.090	103.230	39.46	—	—	—	—
WHAM → GMR	99.663	77.261	115.858	45.88	—	—	—	—

“—” denotes robots unsupported by GMR.

Hardware demonstrations

Real-Robot and Real-Time Teleoperation Performance

Single-arm large-circle motion. All two-stage methods suffer motion collapse.

Hands-on-head squat and leg-lift motion. The two-stage methods produce unnatural squatting and leg-lifting motions.

Leftward side-stretch motion. In the two-stage methods, the right hand fails to extend over the head, while the left arm bends in the wrong direction instead of staying close to the thigh.

Hands-behind-the-back motion. All other two-stage methods incorrectly place both hands in front of the body.

Real-time performance

Streaming inference efficiency

Lower latency / VRAM and higher throughput are better

Method	Latency (ms) ↓	Throughput (FPS) ↑	Peak VRAM (GiB) ↓
Ours	192.8	50.02	7.03
GVHMR → GMR	1369.4	44.31	9.18
GVHMR → NMR	1305.7	44.36	9.73

On the same GPU, we build streaming inference pipelines for different methods and configure each to achieve the lowest possible latency, then measure their median latency and peak VRAM usage. We separately test their maximum throughput under the condition of unlimited input speed.

Streaming Inference Mode

Designed for Real-Time Visual Teleoperation

With BeyondRetarget, responsive real-time monocular visual teleoperation is possible.

Real-time visual teleoperation driven directly by monocular video. Note: The multi-view motion-capture cameras visible in the background are not enabled. This demo relies solely on a single basic 1080p, 30 Hz monocular camera positioned directly in front of the operator.

Citation

BibTeX

The paper citation will be available after the public release.

BeyondRetarget: Learning Executable Humanoid Motions Directly from Monocular Video