SpatialDex: 3D-Grounded Bimanual Dexterity from RGB-Only Human Videos

Abstract

We present SpatialDex, a framework that learns 3D-grounded bimanual dexterity from RGB-only human videos through reconstruction-augmented visuomotor pre-training. The central idea is to learn human behavior together with its spatial context. From each video, we reconstruct the scene geometry and estimate the hand motion, and align them at a shared video-level scale using the hand-size prior of 3D hand estimates. We retain the reconstructed geometry as both a policy input and a prediction target. The policy grounds the token grid of a pre-trained vision-language model in this geometry through gated residual fusion, is trained to predict future scene geometry alongside wrist and finger actions, and generates finger motion conditioned on the predicted trajectories of both wrists. Pre-trained on 1M human-video clips and fine-tuned with a small number of robot demonstrations, SpatialDex outperforms state-of-the-art methods by 16.5 points on nine simulated dexterous tasks across two embodiments and by 38.3 points on six real-world bimanual dexterous tasks. Under a matched comparison, pre-training with reconstructed geometry improves success by 14.2 points over pre-training on hand motion alone.

1M human-video clips for pre-training, reconstructed from RGB-only video.
+38.3 points over the strongest baseline on six real-world bimanual tasks.
+16.5 points over the strongest baseline on nine simulated tasks across two embodiments.
+14.2 points from pre-training with reconstructed geometry instead of hand motion alone.

SpatialDex overview: scene geometry and hand motion reconstructed from RGB-only human videos supervise policy pre-training, and the policy is fine-tuned on robot demonstrations. — **SpatialDex** reconstructs scene geometry and estimates hand motion from RGB-only human videos and aligns them at a shared video-level scale to jointly supervise policy pre-training. The policy maps RGB-D observations to bimanual dexterous actions and is adapted to specific tasks through fine-tuning on robot demonstrations.

How SpatialDex Works

SpatialDex turns each RGB-only human video into paired supervision of scene geometry and hand motion, pre-trains a 3D-grounded visuomotor policy on that supervision, and adapts the policy to a robot with a small number of teleoperated demonstrations.

Aligned Scene and Hand Reconstruction

Data pipeline: hand motion estimation and scene reconstruction from an RGB-only video, joint alignment across frames at a shared scale, producing aligned scene geometry and hand motion. — From an RGB-only video, SpatialDex recovers hand meshes and wrist trajectories, point clouds and camera poses up to an unknown scale. Aligning the reconstructed hand points with the estimated hand meshes across frames yields one scale shared by the whole video and a rigid transformation per frame, which places the scene geometry and the hand motion in a common frame. The aligned reconstruction supplies the depth observations, the wrist and finger action targets, and the future geometry targets for policy pre-training.

3D-Grounded Visuomotor Policy

Policy architecture: depth is back-projected and encoded into a geometry grid aligned with the visual grid, fused through dual sigmoid gates into a VLM backbone, which feeds wrist and finger experts and a training-only geometry predictor. — Reconstructed geometry enters the policy in two roles: as an **input**, fused into the visual token grid of a pre-trained VLM, and as a **target**, predicted at future frames by a training-only head.

Grid-aligned geometry. Depth samples from each image patch are back-projected to metric 3D points, encoded jointly with a point-cloud encoder, and pooled back to their patches, so geometry lands on the same token grid as the VLM's visual features.
Progressive dual-gated fusion. Two sigmoid gates modulate appearance and geometry before a residual update whose weight warms up from zero, leaving the pre-trained visual priors intact early in training.
Forward dynamics prediction. A training-only predictor estimates future scene geometry from the shared context, so the representation must retain spatial structure that the action losses alone do not demand.
Coordinated wrist and finger experts. A wrist expert predicts both wrist trajectories, and a finger expert predicts finger actions conditioned on them.

Dexterous Manipulation on a Real Robot

We evaluate on a bimanual platform with Wuji hands on Tianji arms. For each task we collect 50 teleoperated demonstrations and run 10 trials per method. The clips below show SpatialDex on four scene variations of each task, played at 4× speed.

Pass Flower

Pull the flower from the vase, pass it to the other hand, and place it in the basket.

Pour Water

Lift the kettle and pour water into the pot.

Spray Plant

Pick up the spray bottle and press its trigger to water the plant.

Unzip Bag

Open the zipper of a soft, deformable bag using both hands.

Sweep Trash

Hold the dustpan in one hand and sweep the trash into it with the brush in the other.

Pack Suitcase

Pick up items from the table and pack them into the open suitcase.

Success Rate

SpatialDex (Ours)

81.7

Trained from scratch

ACT

36.7

iDP3

13.3

Pre-trained VLA

π_0.5

43.3

Geometry-aware VLAs

GeoVLA

28.3

Any3D-VLA

30.0

Human-video pre-trained

VITRA

36.7

UniDex

31.7

Success rate (%) over 60 trials per method: six tasks, ten trials each. Hover over or tap a bar for per-task results.

Per-Task Success Rates

Real-world success rates (%). Ten trials per task; the average is over all 60 trials. SpatialDex reaches 81.7%; the strongest baseline, π_0.5, reaches 43.3%.

Method	Pass Flower	Pour Water	Spray Plant	Unzip Bag	Sweep Trash	Pack Suitcase	Avg.
ACT	40.0	40.0	10.0	40.0	40.0	50.0	36.7
iDP3	10.0	10.0	20.0	10.0	0.0	30.0	13.3
π_0.5	40.0	40.0	50.0	50.0	20.0	60.0	43.3
GeoVLA	20.0	20.0	40.0	20.0	30.0	40.0	28.3
Any3D-VLA	30.0	40.0	20.0	20.0	30.0	40.0	30.0
VITRA	30.0	20.0	50.0	40.0	30.0	50.0	36.7
UniDex	30.0	40.0	20.0	40.0	10.0	50.0	31.7
SpatialDex (Ours)	80.0	80.0	90.0	80.0	70.0	90.0	81.7

End-to-End Comparison with Baselines

Full rollouts of SpatialDex, the strongest baseline π_0.5, and the human-video pre-trained VITRA on three of the six tasks, with comparable table setups and played at 4× speed. The percentage next to each method is its success rate over ten trials on that task; the baseline clips show typical failures.