EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videos

Ruihan Yang¹^∗ , Qinxi Yu²^∗ , Yecheng Wu³,⁴ , Rui Yan¹ , Borui Li¹ , An-Chieh Cheng¹ , Xueyan Zou ¹ , Yunhao Fang¹ , Xuxin Cheng¹ , Ri-Zhao Qiu¹ , Hongxu Yin⁴ , Sifei Liu⁴ , Song Han³,⁴ , Yao Lu⁴ , Xiaolong Wang¹ ¹UC San Diego ²UIUC ³MIT ⁴ NVIDIA


Figure 1: EgoVLA. Our vision-language-action model learns manipulation skills from egocentric human videos and transfers them to a bimanual humanoid robot. The top row illustrates the diverse manipulation behaviors demonstrated by humans in the video dataset, while the bottom row shows the robot performing egocentric dexterous manipulation based on the learned skills.

Abstract: Real robot data collection for imitation learning has led to significant advances in robotic manipulation. However, the requirement for robot hardware in the process fundamentally constrains the scale of the data. In this paper, we explore training Vision-Language-Action (VLA) models using egocentric human videos. The benefit of using human videos is not only for their scale but more importantly for the richness of scenes and tasks. With a VLA trained on human video that predicts human wrist and hand actions, we can perform Inverse Kinematics and retargeting to convert the human actions to robot actions. We finetune the model using a few robot manipulation demonstrations to obtain the robot policy, namely EgoVLA. We propose a simulation benchmark called Ego Humanoid Manipulation Benchmark, where we design diverse bimanual manipulation tasks with demonstrations. We fine-tune and evaluate EgoVLA with Ego Humanoid Manipulation Benchmark and show significant improvements over baselines and ablate the importance of human data.Videos can be found on our website: https://rchalyang.github.io/EgoVLA/

1 Introduction

There has been a vast advancement in robotic manipulation in the last few years, thanks to large-scale real robot data collection [\[1,](#page-8-0) [2\]](#page-8-0). Compared to approaches that leverage simulation, directly performing supervised learning with real robot data avoids the Sim2Real domain gap and easily increases the task complexity. To efficiently collect complex robot manipulation data, multiple teleoperation tools with joint mapping [\[3,](#page-8-0) [4,](#page-8-0) [5\]](#page-8-0), exoskeleton [\[6,](#page-8-0) [7,](#page-8-0) [8\]](#page-8-0), and VR devices [\[9,](#page-8-0) [10,](#page-8-0) [11\]](#page-9-0) are proposed. While this is encouraging, the requirement of a robot and an expert operator fundamentally constrains the scale of the data to be collected.

How about learning manipulation from human videos? If we consider human as a special format of robot, there are 8 billion robots continuously operating across the world in all the environments

<span id="page-1-0"></span>

Figure 2: EgoVLA takes visual history, language instruction, and action query token as input. The latent features are converted to human action with the action head. We use the wrist pose and MANO hand parameter[\[18\]](#page-9-0) as human action space.

that we would like robots to operate in. Recent work on Hand-Object Interaction prediction [\[12\]](#page-9-0) has shown promising results in forecasting long-term human intention on manipulation. If we are able to leverage such human data to train a robot policy, it easily scales up not only the number of training data, but more importantly the diversity of tasks and scenes. It allows training on scenes where the current robot might not easily fit in or tasks that are even challenging for teleoperation.

Our key observation is: The difference between human action and robot action spaces might not be that large, and it can be approximated by a few geometric transforms. Instead of training a Robot Vision-Language-Action (VLA) model on robot data [\[13,](#page-9-0) [14,](#page-9-0) [15,](#page-9-0) [16\]](#page-9-0), we propose to train a Human Egocentric VLA (EgoVLA) on human data. Specifically, given a few frames of visual observation language instruction, and current hand pose as input, the VLA will predict human actions in a few steps in the future. The action space includes the human wrist and hand joint angles. This human action space can be converted to a robot action space by converting wrist location to end-effector location through Inverse Kinematics and converting human hand joints to robot hand joints through retargeting. Thus the Human VLA is inherently already a robot policy, except the inputs are human hand images and there is still error in action outputs. We can correct this by further fine-tuning the VLA with a few robot demonstrations collected by teleoperation. In this way, we do not require large-scale robot data for training.

To benchmark robotic manipulation, we propose a new humanoid bimanual manipulation benchmark based on NVIDIA IsaacSim[\[17\]](#page-9-0), namely Ego Humanoid Manipulation Benchmark. In this benchmark, we provide 12 tasks including simple tasks that perform atomic actions and longhorizon tasks that combine multiple atomic actions. We collect 100 demonstrations per task and evaluate models using the benchmark. In our experiments, we first train our EgoVLA on our Ego-Centric Human Manipulation dataset and fine-tune on the humanoid manipulation demonstration collected for specific tasks. In our simulation benchmark, our EgoVLA outperforms different specialist and generalist baselines in both short-horizon and long-horizon tasks and achieves better generalization across visual observation and spatial locations.

2 Related Work

Dexterous Manipulation: Research in dexterous manipulation has progressed from control-based methods [\[19,](#page-9-0) [20,](#page-9-0) [21,](#page-9-0) [22,](#page-9-0) [23,](#page-9-0) [24,](#page-9-0) [25\]](#page-9-0) to learning-driven approaches [\[26,](#page-9-0) [27\]](#page-9-0). While early work emphasized precision, generalization across diverse scenarios remained limited. Learning-based methods introduced pose vector generation [\[28,](#page-9-0) [29,](#page-10-0) [30\]](#page-10-0), intermediate representations [\[31,](#page-10-0) [32\]](#page-10-0), and contact maps [\[33,](#page-10-0) [34\]](#page-10-0), but large-scale dexterous manipulation remains an open challenge. Recent efforts leverage egocentric human videos for training task-specific policies [\[35,](#page-10-0) [36\]](#page-10-0). In contrast, we aim to develop generalist manipulation models directly from egocentric human demonstrations.

Vision-Language-Action (VLA): Vision-Language Models (VLMs)[\[37,](#page-10-0) [38,](#page-10-0) [39\]](#page-10-0) have shown strong generalization across multimodal tasks[\[40,](#page-10-0) [41,](#page-10-0) [42,](#page-10-0) [43,](#page-10-0) [44\]](#page-10-0). Building on this, Vision-Language<span id="page-2-0"></span>Action Models (VLAs)[\[16,](#page-9-0) [14,](#page-9-0) [45,](#page-10-0) [13,](#page-9-0) [15,](#page-9-0) [46\]](#page-11-0) fine-tune VLMs with large-scale robot data, enabling perception-action integration. However, VLA training is data-intensive, often requiring extensive teleoperation[\[47,](#page-11-0) [48\]](#page-11-0) or scripted execution [\[49,](#page-11-0) [50\]](#page-11-0). OpenVLA [\[14\]](#page-9-0) and Octo [\[13\]](#page-9-0) leverage crowdsourced robot datasets [\[1\]](#page-8-0) but face scalability bottlenecks. We propose an alternative: learning policies from human egocentric videos, augmented by small-scale target-domain fine-tuning.

Egocentric Vision: Egocentric vision research [\[51,](#page-11-0) [52,](#page-11-0) [53\]](#page-11-0) traditionally faced limitations in data scale and diversity. Recent datasets [\[54,](#page-11-0) [55\]](#page-11-0) improve coverage but focus on activities beyond current robotic capabilities. Simpler datasets [\[56,](#page-11-0) [57\]](#page-11-0) capture everyday interactions but lack pose annotations. We address this by curating a targeted mixture of datasets and introducing an Egocentric Human Video Dataset (Section 3.1) optimized for dexterous manipulation learning.

Learning from In-the-Wild Video: Several works [\[58,](#page-11-0) [59\]](#page-11-0) propose extracting affordances or interaction cues from in-the-wild videos. Motivated by egocentric vision, recent studies [\[60,](#page-11-0) [61,](#page-11-0) [62,](#page-12-0) [63,](#page-12-0) [64,](#page-12-0) [65,](#page-12-0) [66\]](#page-12-0) pretrain representations using human videos, showing positive transfer. However, most focus on unsupervised learning without leveraging fine-grained hand or wrist pose information. Our work instead uses high-quality egocentric data within the VLA framework to directly improve dexterous policy learning, capitalizing on advances in wearable hand-tracking technologies. TACO HoloAssist HOI4D HOT3D 33,414 51,758 83,509 51,758 65,761 130,457 202,117 118,087 Item 3 Item 4 7 12 Item 5 16 21 Human Video Dataset Split

3 Learning Manipulation Skills from Ego-centric Human Videos

In this section, we describe the construction of our egocentric human manipulation dataset, the training of EgoVLA on this dataset, bridging the embodiment gap between humans and humanoid robots, and the deployment of EgoVLA for manipulation tasks.

3.1 Ego-Centric Human Manipulation Dataset

Human Video Dataset Split 39% 13% TACO HoloAssist HOI4D HOT3D 1 Figure 3: Human Data Following insights from language model and vision-language model training, we emphasize the importance of dataset structure in driving model performance. We construct a large-scale human egocentric manipulation dataset focused on skill-rich video sequences with accompanying pose annotations. The combined dataset contains egocentric RGB observations, wrist poses, hand poses, and camera poses. Our dataset integrates sequences from four sources with their relative proportions shown in Fig. 3: HOI4D consists of 4,000 videos capturing one-handed manipulations such as pick-and-place, reorientation, and articulated object interaction. HOT3D provides 833 minutes of video interacting with 33 rigid objects, with precise 3D hand and camera pose annotations. HoloAssist offers 166 hours of recordings of complex tasks such as battery replacement, furniture assembly, and machine setup. Although its hand pose annotations are noisier, it captures rich bimanual interactions. To avoid overrepresentation of HoloAssist, which has noisier labels, we uniformly sampled 1/10 of it to balance tasks and sources. TACO includes 2,317 motion sequences covering 151 tool-action-object triplets.

Data Processing: Egocentric videos introduce challenges for learning due to continuous camera motion. To mitigate this, we use world-frame camera poses to project future wrist positions into the current camera frame, ensuring consistent supervision. Training samples are generated by sampling RGB observations at 3 FPS, balancing computational efficiency and temporal continuity. In total, our dataset comprises approximately 500,000 image-action pairs across diverse manipulation tasks.

3.2 EgoVLA Model

We build EgoVLA on top of a vision-language model to leverage strong visual and semantic reasoning. Specifically, we use NVILA-2B[\[67\]](#page-12-0) as the backbone for its robust vision-language understanding and compact size, enabling both intention inference and efficient fine-tuning. As shown in Fig[.2,](#page-1-0) EgoVLA takes as input current and historical egocentric visual observations, language instructions, action query tokens, and human proprioception. These inputs are encoded by the VLM backbone and further processed by an action head to predict future human or robot actions.

<span id="page-3-0"></span>

Figure 4: Unified Action Space: MANO hand parameters are used as a shared action space for humans and robots. For robot hands, during training, optimized mano parameters produce the same fingertip position as the robot hand fingertip. A small MLP maps predicted finger tip positions to joint commands during deployment.

Visual Observations consists of six RGB frames: the current observation and five preceding frames, sampled at 0.2sec intervals, covering a 1sec history. Each frame has a resolution of 384 × 384. Language Instructions describe the immediate desired behavior. This design focuses the model on skill execution rather than high-level planning, ensuring a clear mapping between language input and predicted actions. Human Proprioception State includes wrist translations/rotations, and hand pose parameters. These are processed through MLPs before being passed to the action head.

Each predicted action includes the wrist pose (3D translation and rotation in rot6D representation [\[68\]](#page-12-0) in the camera frame) and hand joint angles, represented using the top 15 PCA components of the MANO hand model [\[18\]](#page-9-0). EgoVLA is trained to regress future wrist poses in the camera frame and hand joint parameters. The objective is: L = λwrist transLwrist trans + λwrist rotLwrist rot + λjointLjoint. Lwrist trans and Ljoint are L2 losses for wrist translation and hand joint angle regression. Lwrist rot is a rotation loss for rot6D[\[68\]](#page-12-0) wrist orientation. λjoint, λwrist trans, λwrist rot are weighting coefficients. Full loss details are provided in the Supplementary Materials.

Action Head & Action Query Tokens: The action head is a transformer (300M) consisting of six encoder layers, each with a hidden size of 1536. It takes as input the human (or robot) proprioception state and the latent embeddings corresponding to the action query tokens, and predicts a sequence A^t = [at, at+1, . . . , at+H] over a 1-second horizon (30 future steps at 30 Hz) for both hands. We use the last H = 30 word IDs from the vocabulary as action query tokens.

Training Details: We first pretrain EgoVLA on our Ego-Centric Human Manipulation Dataset for 20 epochs. This is followed by 115 epochs of post-training on robot demonstration data, where the learning rate is reduced after 100 epochs. During training, the full model, including the visual encoder, is fine-tuned.Further training configurations are detailed in the Supplementary Materials.

3.3 Transferring EgoVLA to Humanoid Robot

Humans and humanoid robots share a similar manipulation framework using two arms and hands. However, directly transferring EgoVLA to humanoid robots is challenging due to differences in camera pose, hand morphology, and visual appearance. To enable the deployment, we fine-tune EgoVLA using a small set of robot demonstrations, leveraging a unified action space as in Fig. 4 and we provide more details regarding the Humanoid Robot data in Sec. [4](#page-4-0)

Retargeting robot data to human representation: To fine-tune on robot data, we first align the robot's action space with the human representation. For end-effector poses, 3D transformations align the robot and human coordinate systems. Aligning hand configurations is more complex: we estimate MANO [\[18\]](#page-9-0) parameters that best approximate the robot hand's actuation by minimizing the discrepancy between predicted and observed fingertip positions: minimize L(Θ) =

Θ 5 P⁵ ⁱ=1 SmoothL1(Jpred(Θ)ⁱ , Jobs,i) where Θ ∈ ^R ¹⁵ are the MANO hand parameters, Jpred(Θ) are fingertip positions computed via MANO forward kinematics, and Jobs ∈ ^R 5×3 are the observed robot fingertip positions. This unified action space enables direct fine-tuning of EgoVLA on robot demonstrations without requiring additional architectural changes or reinitialization.

<span id="page-4-0"></span>

Figure 5: Task Visualization. All simulated tasks with predicted wrist trajs from EgoVLA.

Mapping human hand to robot hand: At inference time, the wrist and hand poses predicted by EgoVLA are mapped to the robot's actuators, as illustrated in Fig. [4](#page-3-0) (bottom row). Wrist poses are first converted into robot end-effector poses through 3D transformations, and the corresponding arm joint angles are solved via inverse kinematics (IK). For hand actuation, we use the MANO model to calculate 3D hand keypoints from the predicted MANO parameters. A lightweight MLP then predicts the robot's hand joint commands given 3D hand keypoints. This MLP is trained on robot demonstrations where hand actuations are retargeted into human hand representations. This mapping achieves a mean fingertip position error of 5×10−⁵ , m. Furthermore, replaying raw demonstrations through this retargeting pipeline preserves task validity, indicating that the small errors introduced during retargeting do not significantly affect control performance. Additional implementation details are provided in the Supplementary Materials.

4 Ego Humanoid Manipulation Benchmark

Beyond data scarcity, a major challenge in learning-based robotics is the lack of scalable, robust, and reproducible evaluation. Real-world evaluation is often costly, time-consuming, and raises concerns around safety and reproducibility—barriers that disproportionately affect resource-constrained settings such as academic labs. Recent work [\[69\]](#page-12-0) has shown that simulation-based evaluations are highly correlated with real-world performance, supporting their use as a reliable proxy. To enable consistent benchmarking of humanoid manipulation, we introduce the Ego Humanoid Manipulation Benchmark, built using NVIDIA Isaac Lab [\[70\]](#page-12-0). Ego Humanoid Manipulation Benchmark is not designed for direct sim-to-real transfer. Instead, we leverage simulation, similar to LIBERO [\[71\]](#page-12-0) and SIMPLER [\[69\]](#page-12-0)—as a controlled and reproducible testbed for evaluating manipulation policies. Our simulated platform features the Unitree H1 [\[72\]](#page-12-0) humanoid robot with two Inspire dexterous hands [\[73\]](#page-12-0), and includes 12 manipulation tasks spanning both Short Horizon atomic actions (Push-Box, Flip-Mug, Pour-Balls, Close-Drawer, Open-Drawer, Open-Laptop, Stack-Can) and Long-Horizon, multi-stage skills (Sort-Cans, Insert-Cans, Unload-Cans, Insert-And-Unload-Cans, Stack-Can-Into-Drawer), as shown in Fig. 5.

Observation & Action Space: Our benchmark provides robot joint positions, end-effector poses, contact forces, and egocentric RGB-D visual input for observations. While EgoVLA uses only egocentric vision, end-effector poses, hand joint actuation, and task descriptions, additional modalities are available for future research. The robot is controlled via end-effector control for arms and PD joint control for hands. Each hand has 12 DoFs (6 active, 6 mimic joints). The final 36-dimensional action space combines arm inverse kinematics with direct hand actuation. Control frequency is 30 Hz. We also provide per-step success indicators and sub-task completion flags for every task. Definitions and success metrics for each sub-task are detailed in the supplementary materials.

Diverse Visual Background: Simulation allows full control over visual conditions. We includes 5 room textures (Room 1–5) and 5 table textures (Table 1–5), generating 25 distinct visual background combinations for robust generalization evaluation.

Demonstrations: To support imitation learning, we provide expert demonstrations collected via OpenTelevision [\[10\]](#page-8-0) using the Meta Quest 3. Demonstrations are collected using Room 1, 2, or 3, with Table 1 fixed. For each task, we collect 100 successful demonstrations, with episode lengths ranging from 100 to 500 frames depending on task complexity.

<span id="page-5-0"></span>

Figure 6: Visual Instruction Following. Top: original HOI4D samples with corresponding language instructions. Red lines indicate ground-truth human wrist trajectories, and green lines denote predictions. Bottom: results with modified instructions while keeping the same visual input. EgoVLA adjusts predicted trajectories appropriately according to the new instructions, demonstrating its understanding of language and environment.

Table 1: Evaluation on Short-Horizon Tasks on Seen/Unseen Visual Configurations

5 Experiments

5.1 Human Manipulation Modeling

Before directly evaluating robotic manipulation performance, we first assess how well EgoVLA models human hand motion after training on the Human Ego-Centric Manipulation Dataset. Quantitatively, the average future prediction error for the human wrist translation is approximately 8, cm. When projected onto the 2D image plane, the normalized error is around 0.13, comparable to stateof-the-art results reported in HOI-forecast [\[74\]](#page-12-0). To further examine whether EgoVLA captures environmental context and follows language instructions, we sample examples from the HOI4D evaluation set, modifying the original language prompts while keeping the visual inputs unchanged. As shown in the third column of Fig. 6, when the instruction changes from Put it in the drawer to Take it out of the drawer, the predicted hand trajectories shift from moving into the drawer to moving toward the cabinet surface. Similarly, in the second column, altering the instruction from Put something in it to Open and close the door results in trajectories shifting from placing objects inside the safe to interacting with the safe's door. These results demonstrate that EgoVLA not only accurately models human motion, but also learns the underlying semantic intent behind the actions.

5.2 Humanoid Robot Evaluation

We evaluate the manipulation performance of different models using two metrics: Success Rate (SR), measuring overall task success, and Progress Rate (PSR), defined as the average number of completed subtasks relative to the total subtasks in a long-horizon task.

Baselines: We compare EgoVLA against two baselines: (1) EgoVLA-NoPretrain, which fine-tunes the pre-trained VLM on robot demonstrations without human video pretraining; (2) ACT, which trains a separate specialist transformer for each task individually. For ACT, we adopt the recommended training settings.

Evaluation Setup: We assess model performance under two settings: Seen, where visual backgrounds are identical to those encountered during training, and Unseen, where backgrounds are entirely novel. To promote robustness, object positions are randomized at the start of each roll-

Table 2: Evaluation on Long-Horizon Tasks on Seen/Unseen Visual Configurations.

<span id="page-6-0"></span>

out, with placement constrained to regions not seen during training. Detailed specifications for background splits and object randomization ranges (up to 20cm×20cm) are provided in the supplementary materials. For Seen backgrounds, each model is evaluated with 27 rollouts per task: nine episodes for each of three backgrounds. For Unseen backgrounds, each model is evaluated with 66 rollouts per task: three episodes for each of 22 unseen backgrounds.

We use simulation to ensure reproducible and robust evaluation, but the method itself is modalityagnostic and can learn effectively from any high-quality teleoperated demonstrations, whether collected in simulation or the real world.

EgoVLA Requires Robot Data Post-training: While EgoVLA is pretrained on egocentric human videos using a unified action space, zero-shot deployment on humanoid robots without fine-tuning on robot data results in 0% success across all tasks. This failure stems from appearance, perception, and kinematic mismatches between humans and robots. Even for tasks seen during pretraining (e.g., Pouring Balls), execution fails without robot-specific adaptation, highlighting the need for post-training on in-domain robot data.

Ego-centric Human Pretraining Improves In-Domain Performance: We first evaluate models on Seen visual backgrounds. As shown in Table [1](#page-5-0) and Table 2, EgoVLA consistently outperforms the EgoVLA-NoPretrain baseline across both short- and long-horizon tasks. Gains are especially pronounced on tasks requiring fine-grained manipulation, such as Stack-Cans, Sort-Cans, Insert-And-Unload-Cans, and Flip-Mugs. We attribute this to the human pretraining phase, during which EgoVLA learns general manipulation skills involving "hands" independent of embodiment (human or robot). In contrast, EgoVLA-NoPretrain learns solely from robot demonstrations without such transferable priors. Moreover, the performance gap between EgoVLA and EgoVLA-NoPretrain is larger for long-horizon tasks, achieving approximately 20% higher success rates. Compared to the specialist ACT baselines, generalist models (EgoVLA and EgoVLA-NoPretrain) perform substantially better on both short- and long-horizon tasks. This is likely because specialist models must simultaneously learn low-level manipulation and long-horizon planning from scratch, whereas generalist models leverage shared low-level skills across tasks.

Ego-centric Human Pretraining Enhances Out-of-Domain Generalization: As shown in Table [1,](#page-5-0) EgoVLA maintains strong generalization in short-horizon tasks on Unseen visual backgrounds, with only a minor drop in mean success rate, whereas EgoVLA-NoPretrain exhibits a substantial 23% decline. For long-horizon tasks, as shown in Table 2, EgoVLA achieves around 30% success on unseen backgrounds. Although success rates decline compared to seen environments, the progress rate remains similar, suggesting that failures primarily occur at the final stages of task execution rather than early subgoals. This demonstrates that pretraining on egocentric human videos significantly improves EgoVLA's ability to generalize to novel environments.

Ablation Study — Robot Data Scale: To analyze the contributions of human video pretraining and robot demonstration fine-tuning, we train a variant of EgoVLA using only 50% of the robot demonstrations, denoted as EgoVLA (50%). As shown in Table [1](#page-5-0) and Table 2, while EgoVLA (50%) can still complete some tasks, its overall success rate drops substantially, particularly on long-horizon tasks (from 45.93% to 7.41%). These results suggest that, although pretraining on egocentric hu-


Figure 7: Data Mixture Ablation. EgoVLA pretrained on different mixtures of human egocentric datasets, evaluated on Unseen visual backgrounds for short-horizon tasks. Greater diversity consistently improves generalization performance.



Figure 8: Spatial Distribution. Success rate and progress of EgoVLA under Unseen visual backgrounds, visualized across object spawning positions. The model maintains strong performance across a wide area, with higher success in regions commonly associated with effective bimanual manipulation.

Figure 9: Robot Trajectories (left to right). In the top row, the robot inserts two cans into the tray and unloads them, and in the bottom row, the robot puts the can over the saucer and closes the drawer.

man videos significantly improves both in-domain performance and generalization, EgoVLA still requires a moderate amount of task-specific robot demonstrations to achieve strong performance.

Ablation Study - Pretrain Data Mixture: To further assess the impact of pretraining data composition, we pretrain EgoVLA using different data mixtures, as shown in Fig. 7. Results show that increasing the scale and diversity of human pretraining data consistently improves downstream performance, with both average Success Rate (SR) and Progress Rate (PSR) increasing across tasks. Notably, despite noisy hand annotations in HoloAssist, missing language labels in HOT3D, and limited visual diversity in TACO, we still observe positive transfer, highlighting the robustness of our approach to dataset imperfections.

Performance of EgoVLA Conditioned on Object Spawning Positions: Beyond generalization to diverse visual backgrounds, we further analyze model performance conditioned on object spawning positions. As shown in Fig. 8, we visualize the success rate under Unseen visual backgrounds for short-horizon tasks and the average progress for long-horizon tasks. For short-horizon tasks, we observe that success rates are higher when objects are placed closer to the center of the randomized region. For long-horizon tasks, two distinct high-probability regions emerge. We hypothesize that this arises because many long-horizon tasks involve bimanual operations, with each peak corresponding to successful interactions using either the left or right hand.

Trajectory Visualization: To complement quantitative evaluation, we visualize policy rollouts for long-horizon tasks in Fig. 9. We display the sequence of visual observations alongside the predicted wrist trajectories produced by EgoVLA. For clarity, only the future wrist positions are visualized. Despite training with only 100 demonstrations per task, EgoVLA successfully executes diverse long-horizon tasks while demonstrating strong spatial and visual generalization. Additional trajectory visualizations are provided in the supplementary materials.

6 Conclusion

In this work, we present EgoVLA, a vision-language-action model trained on egocentric human videos for dexterous manipulation. EgoVLA is developed through pretraining a vision-language model on large-scale human egocentric manipulation dataset, and fine-tuning on a small set of robot demonstrations. To enable transfer across embodiments, we introduce a unified action space that aligns human and robot hand representations. Experimental results demonstrate that human video pretraining enables EgoVLA to learn a generalist manipulation policy, achieving strong performance across diverse tasks with limited robot data, with strong generalization ability.

<span id="page-8-0"></span>7 Limitation

Our pretraining framework requires human data with hand and wrist pose annotations, which may limit data availability. However, the increasing accessibility of high-fidelity AR/VR devices (e.g., Quest 3, Vision Pro, Aria Glasses) is expected to ease this constraint. Additionally, although EgoVLA is pretrained with a unified action space, it cannot be directly deployed for manipulation without further fine-tuning on a moderate amount of robot data. Future work may explore improving zero-shot transferability through more embodiment-agnostic pretraining.

References

• [1] Q. Vuong, S. Levine, H. R. Walke, K. Pertsch, and A. S. et al. Open x-embodiment: Robotic learning datasets and RT-x models. In CoRL, 2023. [1,](#page-0-0) [3](#page-2-0) [2] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany,
• M. K. Srirama, L. Y. Chen, K. Ellis, P. D. Fagan, J. Hejna, M. Itkina, M. Lepert, Y. J. Ma,
• P. T. Miller, J. Wu, S. Belkhale, S. Dass, H. Ha, A. Jain, A. Lee, Y. Lee, M. Memmel, S. Park,
• I. Radosavovic, K. Wang, A. Zhan, K. Black, C. Chi, K. B. Hatch, S. Lin, J. Lu, J. Mercat, A. Rehman, P. R. Sanketi, A. Sharma, C. Simpson, Q. Vuong, H. R. Walke, B. Wulfe,
• T. Xiao, J. H. Yang, A. Yavary, T. Z. Zhao, C. Agia, R. Baijal, M. G. Castro, D. Chen, Q. Chen,
• T. Chung, J. Drake, E. P. Foster, J. Gao, D. A. Herrera, M. Heo, K. Hsu, J. Hu, D. Jackson,
• C. Le, Y. Li, K. Lin, R. Lin, Z. Ma, A. Maddukuri, S. Mirchandani, D. Morton, T. Nguyen,
• A. O'Neill, R. Scalise, D. Seale, V. Son, S. Tian, E. Tran, A. E. Wang, Y. Wu, A. Xie, J. Yang,
• P. Yin, Y. Zhang, O. Bastani, G. Berseth, J. Bohg, K. Goldberg, A. Gupta, A. Gupta, D. Jayaraman, J. J. Lim, J. Malik, R. Mart´ın-Mart´ın, S. Ramamoorthy, D. Sadigh, S. Song, J. Wu,
• M. C. Yip, Y. Zhu, T. Kollar, S. Levine, and C. Finn. Droid: A large-scale in-the-wild robot manipulation dataset. 2024. [1](#page-0-0) [3] A. Iyer, Z. Peng, Y. Dai, I. Guzey, S. Haldar, S. Chintala, and L. Pinto. Open teach: A versatile teleoperation system for robotic manipulation. arXiv preprint arXiv:2403.07870, 2024. [1](#page-0-0) [4] S. Dass, W. Ai, Y. Jiang, S. Singh, J. Hu, R. Zhang, P. Stone, B. Abbatematteo, and R. Martin-Martin. Telemoma: A modular and versatile teleoperation system for mobile manipulation. arXiv preprint arXiv:2403.07869, 2024. [1](#page-0-0) [5] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. RSS, 2023. [1,](#page-0-0) [6,](#page-5-0) [7,](#page-6-0) [18](#page-17-0) [6] L. Zhao, T. Yang, Y. Yang, and P. Yu. A wearable upper limb exoskeleton for intuitive teleoperation of anthropomorphic manipulators. Machines, 11(4):441, 2023. [1](#page-0-0) [7] H. Fang, H.-S. Fang, Y. Wang, J. Ren, J. Chen, R. Zhang, W. Wang, and C. Lu. Airexo: Lowcost exoskeletons for learning whole-arm manipulation in the wild. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 15031–15038. IEEE, 2024. [1](#page-0-0) [8] S. Yang, M. Liu, Y. Qin, D. Runyu, L. Jialong, X. Cheng, R. Yang, S. Yi, and X. Wang. Ace: A cross-platfrom visual-exoskeletons for low-cost dexterous teleoperation. arXiv preprint arXiv:240, 2024. [1](#page-0-0) [9] A. Naceri, D. Mazzanti, J. Bimbo, Y. T. Tefera, D. Prattichizzo, D. G. Caldwell, L. S. Mattos, and N. Deshpande. The vicarios virtual reality interface for remote robotic teleoperation: Teleporting for intuitive tele-manipulation. Journal of Intelligent & Robotic Systems, 101: 1–16, 2021. [1](#page-0-0) [10] X. Cheng, J. Li, S. Yang, G. Yang, and X. Wang. Open-television: Teleoperation with immersive active visual feedback. arXiv preprint arXiv:2407.01512, 2024. [1,](#page-0-0) [5](#page-4-0)

• <span id="page-9-0"></span>[11] R. Ding, Y. Qin, J. Zhu, C. Jia, S. Yang, R. Yang, X. Qi, and X. Wang. Bunny-visionpro: Realtime bimanual dexterous teleoperation for imitation learning. 2024. URL https://arxiv. org/abs/2407.03162. [1](#page-0-0) [12] J. Tian, L. Yang, R. Ji, Y. Ma, L. Xu, J. Yu, Y. Shi, and J. Wang. Gaze-guided hand-object interaction synthesis: Benchmark and method. arXiv preprint arXiv:2403.16169, 2024. [2](#page-1-0) [13] Octo Model Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna,
• C. Xu, J. Luo, T. Kreiman, Y. Tan, L. Y. Chen, P. Sanketi, Q. Vuong, T. Xiao, D. Sadigh,
• C. Finn, and S. Levine. Octo: An open-source generalist robot policy. In RSS, 2024. [2,](#page-1-0) [3](#page-2-0) [14] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. P. Foster, P. R. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine,
• P. Liang, and C. Finn. OpenVLA: An open-source vision-language-action model. In CoRL, 2024. [2,](#page-1-0) [3](#page-2-0) [15] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman,
• B. Ichter, et al. π0: A vision-language-action flow model for general robot control. arXiv preprint, 2024. [2,](#page-1-0) [3](#page-2-0) [16] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess,
• A. Dubey, C. Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint, 2023. [2,](#page-1-0) [3](#page-2-0) [17] NVIDIA. Isaac sim: Advanced simulation for robotics development, 2024. Accessed: Jun 2024. [2](#page-1-0) [18] J. Romero, D. Tzionas, and M. J. Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 36(6), Nov. 2017. [2,](#page-1-0) [4,](#page-3-0) [15,](#page-14-0) [18](#page-17-0) [19] A. Rodriguez, M. T. Mason, and S. Ferry. From caging to grasping. IJRR, 2012. [2](#page-1-0) [20] C. Rosales, R. Suarez, M. Gabiccini, and A. Bicchi. On the synthesis of feasible and prehensile ´ robotic grasps. In ICRA, 2012. [2](#page-1-0) [21] D. Prattichizzo, M. Malvezzi, M. Gabiccini, and A. Bicchi. On the manipulability ellipsoids of underactuated robotic hands with compliance. RAS, 2012. [2](#page-1-0) [22] J. Ponce, S. Sullivan, J.-D. Boissonnat, and J.-P. Merlet. On characterizing and computing three-and four-finger force-closure grasps of polyhedral objects. In ICRA, 1993. [2](#page-1-0) [23] J. Ponce, S. Sullivan, A. Sudsang, J.-D. Boissonnat, and J.-P. Merlet. On computing four-finger equilibrium and force-closure grasps of polyhedral objects. IJRR, 1997. [2](#page-1-0) [24] Y. Zheng and C.-M. Chew. Distance between a point and a convex cone in n-dimensional space: Computation and applications. T-RO, 2009. [2](#page-1-0) [25] H. Dai, A. Majumdar, and R. Tedrake. Synthesis and optimization of force closure grasps via sequential semidefinite programming. ISRR, 2018. [2](#page-1-0) [26] O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki,
• A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. IJRR, 2020. [2](#page-1-0) [27] A. Nagabandi, K. Konolige, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. In CoRL, 2020. [2](#page-1-0) [28] H. Jiang, S. Liu, J. Wang, and X. Wang. Hand-object contact consistency reasoning for human grasps generation. In ICCV, 2021. [2](#page-1-0)

• <span id="page-10-0"></span>[29] E. Corona, A. Pumarola, G. Alenya, F. Moreno-Noguer, and G. Rogez. Ganhand: Predicting human grasp affordances in multi-object scenes. In CVPR, 2020. [2](#page-1-0) [30] L. Yang, X. Zhan, K. Li, W. Xu, J. Li, and C. Lu. Cpf: Learning a contact potential field to model the hand-object interaction. In ICCV, 2021. [2](#page-1-0) [31] L. Shao, F. Ferreira, M. Jorda, V. Nambiar, J. Luo, E. Solowjow, J. A. Ojea, O. Khatib, and
• J. Bohg. Unigrasp: Learning a unified model to grasp with multifingered robotic hands. RA-L, 2020. [2](#page-1-0) [32] A. Wu, M. Guo, and C. K. Liu. Learning diverse and physically feasible dexterous grasps with generative model and bilevel optimization. CoRL, 2022. [2](#page-1-0) [33] S. Brahmbhatt, A. Handa, J. Hays, and D. Fox. Contactgrasp: Functional multi-finger grasp synthesis from contact. In IROS, 2019. [2](#page-1-0) [34] D. Turpin, L. Wang, E. Heiden, Y.-C. Chen, M. Macklin, S. Tsogkas, S. Dickinson, and
• A. Garg. Grasp'd: Differentiable contact-rich grasp synthesis for multi-fingered hands. In ECCV, 2022. [2](#page-1-0) [35] R.-Z. Qiu, S. Yang, X. Cheng, C. Chawla, J. Li, T. He, G. Yan, D. J. Yoon, R. Hoque,
• L. Paulsen, G. Yang, J. Zhang, S. Yi, G. Shi, and X. Wang. Humanoid policy human policy, 2025. URL https://arxiv.org/abs/2503.13441. [2](#page-1-0) [36] S. Kareer, D. Patel, R. Punamiya, P. Mathur, S. Cheng, C. Wang, J. Hoffman, and D. Xu. Egomimic: Scaling imitation learning via egocentric video, 2024. URL https://arxiv. org/abs/2410.24221. [2](#page-1-0) [37] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint, 2023. [2](#page-1-0) [38] J. Lin, H. Yin, W. Ping, P. Molchanov, M. Shoeybi, and S. Han. Vila: On pre-training for visual language models. In CVPR, 2024. [2,](#page-1-0) [6,](#page-5-0) [7](#page-6-0) [39] S. Tong, E. Brown, P. Wu, S. Woo, M. Middepogu, S. C. Akula, J. Yang, S. Yang, A. Iyer,
• X. Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint, 2024. [2](#page-1-0) [40] S. Pratt, I. Covert, R. Liu, and A. Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. In ICCV, 2023. [2](#page-1-0) [41] Y. Alaluf, E. Richardson, S. Tulyakov, K. Aberman, and D. Cohen-Or. Myvlm: Personalizing vlms for user-specific queries. In ECCV, 2025. [2](#page-1-0) [42] W. Kuo, Y. Cui, X. Gu, A. Piergiovanni, and A. Angelova. Open-vocabulary object detection upon frozen vision and language models. In ICLR, 2023. [2](#page-1-0) [43] D.-A. Huang, S. Liao, S. Radhakrishnan, H. Yin, P. Molchanov, Z. Yu, and J. Kautz. Lita: Language instructed temporal-localization assistant. In ECCV, 2025. [2](#page-1-0) [44] T. Lv, Y. Huang, J. Chen, Y. Zhao, Y. Jia, L. Cui, S. Ma, Y. Chang, S. Huang, W. Wang, et al. Kosmos-2.5: A multimodal literate model. arXiv preprint, 2023. [2](#page-1-0) [45] J. Wen, Y. Zhu, J. Li, M. Zhu, K. Wu, Z. Xu, R. Cheng, C. Shen, Y. Peng, F. Feng, et al. Tinyvla: Towards fast, data-efficient vision-language-action models for robotic manipulation. arXiv preprint, 2024. [3](#page-2-0)

• <span id="page-11-0"></span>[46] P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi,
• C. Finn, N. Fusai, M. Y. Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak,
• T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke,
• A. Walling, H. Wang, L. Yu, and U. Zhilinsky. π0.5: a vision-language-action model with open-world generalization, 2025. URL https://arxiv.org/abs/2504.16054. [3](#page-2-0) [47] A. Mandlekar, Y. Zhu, A. Garg, J. Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta,
• E. Orbay, et al. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In CoRL, 2018. [3](#page-2-0) [48] A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y. Narang, L. Fan, Y. Zhu, and D. Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. In CoRL, 2023. [3](#page-2-0) [49] S. Dasari, F. Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, S. Levine, and
• C. Finn. Robonet: Large-scale multi-robot learning. In CoRL, 2019. [3](#page-2-0) [50] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly,
• M. Kalakrishnan, V. Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In CoRL, 2018. [3](#page-2-0) [51] D. Damen, H. Doughty, G. M. Farinella, A. Furnari, E. Kazakos, J. Ma, D. Moltisanti,
• J. Munro, T. Perrett, W. Price, et al. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. IJCV, 2022. [3](#page-2-0) [52] G. A. Sigurdsson, A. K. Gupta, C. Schmid, A. Farhadi, and A. Karteek. Actor and observer: Joint modeling of first and third-person videos. In CVPR, 2018. [3](#page-2-0) [53] Y. Li, Z. Ye, and J. M. Rehg. Delving into egocentric actions. In CVPR, 2015. [3](#page-2-0) [54] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang,
• M. Liu, X. Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, 2022. [3](#page-2-0) [55] K. Grauman, A. Westbury, L. Torresani, K. Kitani, J. Malik, T. Afouras, K. Ashutosh,
• V. Baiyya, S. Bansal, B. Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In CVPR, 2024. [3](#page-2-0) [56] F. Mahdisoltani, G. Berger, W. Gharbieh, D. Fleet, and R. Memisevic. On the effectiveness of task granularity for transfer learning. arXiv preprint, 2018. [3](#page-2-0) [57] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti,
• J. Munro, T. Perrett, W. Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In ECCV, 2018. [3](#page-2-0) [58] S. Bahl, R. Mendonca, L. Chen, U. Jain, and D. Pathak. Affordances from human videos as a versatile representation for robotics. 2023. [3](#page-2-0) [59] C. Wen, X. Lin, J. So, K. Chen, Q. Dou, Y. Gao, and P. Abbeel. Any-point trajectory modeling for policy learning, 2023. [3](#page-2-0) [60] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual representation for robot manipulation, 2022. URL https://arxiv.org/abs/2203.12601. [3](#page-2-0) [61] A. Majumdar, K. Yadav, S. Arnaud, Y. J. Ma, C. Chen, S. Silwal, A. Jain, V.-P. Berges,
• T. Wu, J. Vakil, P. Abbeel, J. Malik, D. Batra, Y. Lin, O. Maksymets, A. Rajeswaran, and
• F. Meier. Where are we in the search for an artificial visual cortex for embodied intelligence? In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=6qLzQeFGio. [3](#page-2-0)

• <span id="page-12-0"></span>[62] S. Karamcheti, S. Nair, A. S. Chen, T. Kollar, C. Finn, D. Sadigh, and P. Liang. Languagedriven representation learning for robotics. In Robotics: Science and Systems (RSS), 2023. [3](#page-2-0) [63] J. Yang, B. Liu, J. Fu, B. Pan, G. Wu, and L. Wang. Spatiotemporal predictive pre-training for robotic motor control, 2024. URL https://arxiv.org/abs/2403.05304. [3](#page-2-0) [64] J. Zeng, Q. Bu, B. Wang, W. Xia, L. Chen, H. Dong, H. Song, D. Wang, D. Hu, P. Luo, et al. Learning manipulation by predicting interaction. arXiv preprint arXiv:2406.00439, 2024. [3](#page-2-0) [65] S. Ye, J. Jang, B. Jeon, S. Joo, J. Yang, B. Peng, A. Mandlekar, R. Tan, Y.-W. Chao, B. Y. Lin,
• L. Liden, K. Lee, J. Gao, L. Zettlemoyer, D. Fox, and M. Seo. Latent action pretraining from videos, 2024. URL https://arxiv.org/abs/2410.11758. [3](#page-2-0) [66] W. Lirui, C. Xinlei, Z. Jialiang, and H. Kaiming. Scaling proprioceptive-visual learning with heterogeneous pre-trained transformers. In Neurips, 2024. [3](#page-2-0) [67] Z. Liu, L. Zhu, B. Shi, Z. Zhang, Y. Lou, S. Yang, H. Xi, S. Cao, Y. Gu, D. Li, X. Li, Y. Fang,
• Y. Chen, C.-Y. Hsieh, D.-A. Huang, A.-C. Cheng, V. Nath, J. Hu, S. Liu, R. Krishna, D. Xu,
• X. Wang, P. Molchanov, J. Kautz, H. Yin, S. Han, and Y. Lu. Nvila: Efficient frontier visual language models, 2025. URL https://arxiv.org/abs/2412.04468. [3](#page-2-0) [68] Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li. On the continuity of rotation representations in neural networks, 2020. URL https://arxiv.org/abs/1812.07035. [4,](#page-3-0) [17](#page-16-0) [69] X. Li, K. Hsu, J. Gu, K. Pertsch, O. Mees, H. R. Walke, C. Fu, I. Lunawat, I. Sieh, S. Kirmani, S. Levine, J. Wu, C. Finn, H. Su, Q. Vuong, and T. Xiao. Evaluating real-world robot manipulation policies in simulation, 2024. URL https://arxiv.org/abs/2405.05941. [5](#page-4-0) [70] M. Mittal, C. Yu, Q. Yu, J. Liu, N. Rudin, D. Hoeller, J. L. Yuan, R. Singh, Y. Guo, H. Mazhar,
• A. Mandlekar, B. Babich, G. State, M. Hutter, and A. Garg. Orbit: A unified simulation framework for interactive robot learning environments. IEEE Robotics and Automation Letters, 8(6):3740–3747, 2023. doi:10.1109/LRA.2023.3270034. [5](#page-4-0) [71] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning, 2023. URL https://arxiv.org/abs/2306. 03310. [5](#page-4-0) [72] U. Robotics. H1, 2024. Accessed: Sep 2024. [5](#page-4-0) [73] I. Robots. The dexterous hands, 2024. Accessed: Jul 2024. [5](#page-4-0) [74] S. Liu, S. Tripathi, S. Majumdar, and X. Wang. Joint hand motion and interaction hotspots prediction from egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. [6](#page-5-0) [75] X. Wang, T. Kwon, M. Rad, B. Pan, I. Chakraborty, S. Andrist, D. Bohus, A. Feniello, B. Tekin,
• F. V. Frujeri, N. Joshi, and M. Pollefeys. Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 20270–20281, October 2023. [15,](#page-14-0) [16](#page-15-0) [76] Y. Liu, Y. Liu, C. Jiang, K. Lyu, W. Wan, H. Shen, B. Liang, Z. Fu, H. Wang, and L. Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21013–21022, June 2022. [15,](#page-14-0) [16,](#page-15-0) [18,](#page-17-0) [19,](#page-18-0) [22](#page-21-0) [77] Y. Liu, H. Yang, X. Si, L. Liu, Z. Li, Y. Zhang, Y. Liu, and L. Yi. Taco: Benchmarking generalizable bimanual tool-action-object understanding. arXiv preprint arXiv:2401.08399, 2024. [15,](#page-14-0) [16](#page-15-0)

• <span id="page-13-0"></span>[78] P. Banerjee, S. Shkodrani, P. Moulon, S. Hampali, F. Zhang, J. Fountain, E. Miller, S. Basol,
• R. Newcombe, R. Wang, J. J. Engel, and T. Hodan. Introducing hot3d: An egocentric dataset for 3d hand and object tracking. arXiv preprint arXiv:2406.09598, 2024. [15,](#page-14-0) [18](#page-17-0) [79] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez,
• D. Haziza, F. Massa, A. El-Nouby, R. Howes, P.-Y. Huang, H. Xu, V. Sharma, S.-W. Li,
• W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, J. Mairal,
• P. Labatut, A. Joulin, and P. Bojanowski. Dinov2: Learning robust visual features without supervision, 2023. [17](#page-16-0) [80] T. Darcet, M. Oquab, J. Mairal, and P. Bojanowski. Vision transformers need registers, 2023. [17](#page-16-0) [81] NVIDIA. Nvidia omniverse: A platform for virtual collaboration and real-time simulation, 2024. Accessed: Jun 2024. [22](#page-21-0)

Appendix

<span id="page-14-0"></span>1 Dataset Details

Language Label: The combined dataset includes ego-centric RGB visual observations, wrist poses, hand poses, and camera poses. Notably, HoloAssist, TACO and HOI4D provide specific language labels for each video clip, while HOT3D lacks language annotations. To maintain consistency across datasets, we include placeholder language instructions for HOT3D.

MANO Shape Parameters: To streamline the problem, we use the average human hand shape provided by MANO [\[18\]](#page-9-0), focusing on manipulation skills learning from human demonstrations.

Hand Pose Parameters: For ego-centric video dataset (HoloAssist[\[75\]](#page-12-0)) that utilize hand model other than MANO[\[18\]](#page-9-0), we retarget the provided hand poses to the MANO [\[18\]](#page-9-0) representation, ensuring unified modeling.

Section 3.1 presents an overview of our Ego-Centric Human Video Dataset. Here, we provide a detailed visualization of the task distribution for HOI4D [\[76\]](#page-12-0) and HoloAssist [\[75\]](#page-12-0), as shown in Fig. [10.](#page-15-0) The task distribution for HOT3D[\[78\]](#page-13-0) is not visualized, as task labels are not provided for its sequences.

For HOI4D [\[76\]](#page-12-0), we show the task descriptions across all sequences, while for HoloAssist [\[75\]](#page-12-0) and TACO [\[77\]](#page-12-0), we illustrate the verbs from the action descriptions. To enhance clarity, a logarithmic scale is applied to the frequency distributions for both datasets. As shown in Fig. [10,](#page-15-0) all datasets provide ego-centric human video across diverse tasks.

<span id="page-15-0"></span>

(a) HOI4D Task distribution


(b) Taco Task distribution


(c) HoloAssist distribution

Figure 10: Human Video Distribution. We visualize the human video task distribution over all HOI4D [76], HoloAssist [75] and TACO [77] sequences.

<span id="page-16-0"></span>2 Training Details

2.1 EgoVLA Training Details

We provide detailed training configurations to ensure the faithful reproduction of our results. All models were trained using 32 A100 GPUs.

Training Objectives: Our full training objective is defined as follows:

$$\mathcal{L} = \lambda_{\text{wrist trans}} \mathcal{L}_{\text{wrist trans}} + \lambda_{\text{wrist rot}} \mathcal{L}_{\text{wrist rot}} + \lambda_{\text{joint}} \mathcal{L}_{\text{joint}}$$

where Lwrist trans = ∥Tpred − Tgt∥ 2 2 is L2 loss for wrist translation regression. Tpred is the predicted wrist translation in the camera frame, and Pgt is the ground-truth wrist translation. Lwrist rot = ∥Rpred − Rgt∥ 2 2 is the rotation loss for wrist rotation prediction where Rpred is the predicted wrist rotation in the camera frame, and Rgt is the ground-truth wrist rotation. Our model predicts rotation in rot6d[\[68\]](#page-12-0), and we convert the predicted rotation to rotation matrix before compute the loss. Ljoint = ∥Θpred − Θgt∥ 2 2 is L2 loss for hand joint angle regression, where Θpred are the predicted MANO parameters, and Θgt are the ground-truth MANO parameters. λjoint, λwrist trans, λwrist rot are weights to balance different terms.

We use the following hyperparameters for training on our ego-centric human video dataset and during fine-tuning with robot demonstrations (as in Table 3).

Table 3: Hyper Parameters for Pretraining on Human Videos

2.2 ACT Training Details

All ACT models were trained using 3 NVIDIA RTX A4000. To obtain a more thorough assessment of performance on ACT, we trained ACT policies for each task with new settings. We replace the visual backbone with DinoV2 [\[79\]\[80\]](#page-13-0). Rather than padding episodes with the last frame, we modified the dataloader to randomly select an episode and a segment from the dataset. We set action chunk size to 64 for short horizon task Close-Drawer and 128 for the others. We use the following hyperparameters for training ACT models(as in Table 4).

Table 4: Hyper Parameters for ACT models

<span id="page-17-0"></span>3 Detailed Action Formulation

3.1 Inference

As described in Section 3, our EgoVLA generates H = 30 actions per timestep at a frequency of 30 Hz, predicting actions for a 1-second future window. When deployed on the humanoid robot in simulation, the policy operates at 30 Hz. To enhance the smoothness of the robot's behavior and incorporate long-term planning capabilities, we employ action-chunking as described in ALOHA [\[5\]](#page-8-0). For smoothing, we use a parameter value of 0.8.

3.2 Detailed MANO Hand Parameterization

The MANO hand model [\[18\]](#page-9-0) includes 15 ball joints with 45 degrees of freedom (DOFs). However, the human hand lacks this level of flexibility. To address this, MANO provides a low-rank representation using PCA. Consistent with HOI4D [\[76\]](#page-12-0) and HOT3D [\[78\]](#page-13-0), we adopt the first 15 principal components of the PCA representation for hand pose parameterization, effectively representing the action space in a compact yet expressive form.

3.3 Retargeting during deployment

To ensure consistency between training and inference, we train a compact MLP for retargeting. Specifically, this retargeting MLP takes as input the 3D fingertip positions in the wrist frame of both hands and outputs the actuation values for all degrees of freedom (DOFs) of both hands.

The retargeting MLP is a four-layer neural network with hidden layer sizes of [64, 128, 64]. Its training data is derived from the process of converting robot demonstrations into human hand representations, as detailed in Sec.3. By employing this retargeting MLP, we not only achieve consistent retargeting results across training and inference but also enhance inference speed.

Table 5: Hyper Parameters for Retargeting MLP

<span id="page-18-0"></span>4 Human Prediction Visualization

In addition to the instruction-following results presented in Sec.5, this section provides additional visualizations of human trajectory predictions. We showcase the predicted future hand trajectories alongside the ground truth trajectories in Fig. 11. As illustrated, our EgoVLA produces accurate and consistent predictions across diverse scenes, object instances, and tasks. All these human videos are never seen by the model during training.

In Fig. 11a, the hand trajectory shows the approach to the knife, followed by lifting it and cutting the apple. Fig.11b and Fig.11c depict clear hand movements along the door's rotational axis during operation. In Fig. 11g, our model accurately predicts the placement position within the drawer. Lastly, in Fig.11d and Fig.11e, the predictions remain consistent across different kettle instances, indicating the model's ability to understand the shared semantics of similar objects.



Figure 11: Human Trajectory Prediction: The green dotted lines are the future human hand trajectories predicted by our EgoVLA and the red dotted lines are the ground-truth future human hand trajectories. All sequences are from our evaluation set built upon HOI4D [\[76\]](#page-12-0) dataset. The language sequence descriptions are also provided as captions for each sequence.

5 Policy Trajectories Visualization

In addition to the policy trajectories discussed in Sec.5, this section provides additional visualizations of policy rollouts for various tasks. We present short-horizon task trajectories in Fig.12 and Fig.12. These figures highlight our model's ability to perform fundamental manipulation skills, including: Pushing: Illustrated in Fig. 12b. Grasping and Placing: Shown in Fig. 12d.

In Fig. [13,](#page-20-0) our model demonstrates its capacity to integrate these atomic skills from short-horizon tasks to solve diverse long-horizon tasks using a single generalist policy. For example: In Fig. [13d,](#page-20-0) the policy successfully executes a long-horizon task involving picking up a can, stacking the on the saucer and closing the drawer. In Fig. [13e,](#page-20-0) the policy achieves a sequence of actions to sort four different cans. This involves picking up each can and placing it in the correct container consecutively, demonstrating the model's accuracy and consistency in long-horizon tasks.


Figure 12: Robot Trajectories for Short-Horizon Tasks. We visualize the policy trajectories for short-horizon tasks in our Ego Humanoid Manipulation Benchmark. The green dotted lines are the future robot hand trajectories predicted by our EgoVLA.

<span id="page-20-0"></span>

Figure 13: Robot Trajectories for Long-Horizon Tasks. We visualize the policy trajectories for long-horizon tasks in our Ego Humanoid Manipulation Benchmark. The green dotted lines are the future robot hand trajectories predicted by our EgoVLA.

<span id="page-21-0"></span>6 Humanoid Manipulation Benchmark Details

6.1 Assets

The asset library combines models from NVIDIA Omniverse [\[81\]](#page-13-0) and 3D-scanned meshes from HOI4D [\[76\]](#page-12-0), with modifications (e.g., added handles) to improve manipulability. We also fine-tune physics parameters to ensure realistic interactions while maintaining simulation efficiency.

6.2 Detailed Task Lists and Language Instruction for Every Task

• Pour-Balls: "Pour balls in cup into bowl"
• Push-Box: "Push box to the marker"
• Close-Drawer: "Close the opened drawer"
• Open-Drawer: "Open the closed drawer"
• Flip-Mug: "Flip the mug"
• Open-Laptop: "Open the laptop"
• Stack-Can: "Put can on the saucer"
• Sort-Cans: "Put sprite cans to the left box, and orange cans to the right box"
• Insert-Cans: "Insert cans into the boxes"
• Insert-And-Unload-Cans: "Insert the right can into the slot and insert the left can into the slot, unload the right cans and then unload the left cans"
• Unload-Cans: "Unload the right cans and then unload the left cans"
• Stack-Can-Into-Drawer: "Put can on the saucer, and Close the drawer"

6.3 Object Randomization Range Visualization

We provide the visualization for range of object randomization. Objects are randomized within a rectangle. The visualization of object at upper left and bottom right corners is shown in Fig. [14.](#page-22-0)

6.4 Environment Background Visualization

We provide a visualization of the diverse background of the environments. In total, we include 5 rooms and 5 tables, which can be combined in various configurations. The visualization of rooms and tables is shown in Fig. [15.](#page-22-0)

6.5 Training & Evaluation Visual Configuration Visualization

We provide a visualization of the training and evaluation visual configuration. As shown in Fig. [16](#page-23-0) and discussed in the main paper, we collect and train our models with three visual configurations and evaluate models on 22 unseen visual configurations.

6.6 Simulation Details and Success & Subtask definitions

To maintain a control frequency of 30 Hz, the simulation decimation and render interval are set based on the time step (dt), calculated as:

simulation decimation = render interval = $$\frac{1}{30 * dt}$$

For each environment, the detailed physics parameters and task definitions are provided below:

• Push-Box
• Physics Parameters:

<span id="page-22-0"></span>

(a) Close-Drawer: Drawer randomized within 0.2 x

0.1 (meters)

(b) Open-Drawer: Drawer randomized within 0.2 x

0.1

(c) Flip-Mug: Mug randomized within 0.2 x 0.2 (d) Pour-Balls: Bowl randomized within 0.2 x 0.2

(e) Insert-Cans: Cans randomized within 0.12 x 0.12 (f) Insert-And-Unload-Cans: Cans randomized

within 0.12 x 0.12

(g) Unload-Cans: Cans and container randomized

within 0.2 x 0.1

(h) Open-Laptop: Laptop randomized within 0.2 x

0.2

(i) Sort-Cans: Cans randomized within 0.12 x 0.12 (j) Push-Box: Box randomized within 0.2 x 0.2; Goal

randomized within 0.06 x 0.2

(k) Stack-Can: Can and plate randomized within 0.2

x 0.2

(l) Stack-Can-Into-Drawer: Can randomized within 0.2 x 0.1; Plate and drawer fixed in vertical direction and randomized within 0.2 along horizontal axis

Figure 14: Object Randomization Range


Figure 15: Diverse Environment Background: we included 5 different rooms and 5 different tables resulting 25 different visual configurations

<span id="page-23-0"></span>

Figure 16: Training & Evaluation Configuration

• \* Physics simulation time-step in seconds (dt): ¹ 120 \* Box contact offset (in meters): 0.02
• Task Definition: \ Final task success if distance between box and goal marker is smaller than 0.08. \ Subtask reach success if distance between end effector(EE) and box is smaller than 0.13.
• Flip-Mug
• Physics Parameters: \ dt: ¹ 240 \ Mug collision approximation: SDF Mesh \* Mug contact offset: 0.002
• Task Definition: \ Final task success if z unit vector(unit vector in world frame pointing upwards) transformed by mug frame has z value greater than 0.5. \ Subtask reach success if distance between EE and mug is smaller than 0.12.
• Pour-Balls
• Physics Parameters: \ dt: ¹ 240 \ Mug collision approximation: SDF Mesh \ Mug contact offset: 0.002 \ Ball contact offset: 0.002
• Task Definition: \ Final task success if at least 3 balls are within the mug \ Subtask reach success if distance between EE and bottle is smaller than 0.12. \* Subtask lift success if height of bottle increases by at least 0.06.
• Sort-Cans
• Physics Parameters: \ dt: ¹ 120 \ Can collision approximation: Convex hull \* Mug contact offset: 0.0015

• \ Container collision approximation: Convex Decomposition \ Container contact offset: 0.0015
• Task Definition: \ Final task success if both Sprite cans are inside the left container and both Fanta cans are inside the right container \ Subtask reaches success if the distance between EE and can is smaller than 0.12. Value of subtask reach represents the number of cans that satisfy the condition. \ Subtask lift success if height of the can increases by at least 0.06. Value of the subtask lift represents the number of cans that satisfy the condition. \ Subtask sort represents the number of cans that are inside the correct container.
• Insert-Cans
• Physics Parameters: \ dt: ¹ 120 \ Can collision approximation: Convex hull \ Can contact offset: 0.0015 \ Container collision approximation: SDF Mesh \* Container contact offset: 0.002
• Task Definition: \ Final task success if cans x,y coordinates are within the container and the z value is smaller than the threshold for inserted cans (1.065) \ Subtask reaches success if the distance between EE and can is smaller than 0.12. Value of subtask reach represents the number of cans that satisfy the condition. \ Subtask lift success if height of the can increases by at least 0.06. Value of the subtask lift represents the number of cans that satisfy the condition. \ Subtask insert represents the number of cans that are inserted.
• Unload-Cans
• Physics Parameters: \ dt: ¹ 120 \ Can collision approximation: Convex hull \ Can contact offset: 0.0015 \ Container collision approximation: SDF Mesh \* Container contact offset: 0.002
• Task Definition: \ Final task success if cans x,y coordinates are not within the container, and can is standing upwards on the table (same calculation as Flip-Mug), and height of can is greater than table surface \ Subtask reach success if distance between EE and can is smaller than 0.12. Value of subtask reach represents the number of cans that satisfy the condition. \ Subtask lift success if height of can increases by at least 0.06. Value of subtask lift represents the number of cans that satisfy the condition. \ Subtask unload represents the number of cans that are unloaded.
• Insert-And-Unload-Cans
• Physics Parameters: \ dt: ¹ 120 \ Can collision approximation: Convex hull \ Can contact offset: 0.0015 \ Container collision approximation: SDF Mesh \* Container contact offset: 0.002
• Task Definition:

• \ Final task success if cans are inserted first then unloaded \ Subtask reach success if distance between EE and can is smaller than 0.12. Value of subtask reach represents the number of cans that satisfy the condition. \ Subtask lift success if height of can increases by at least 0.06. Value of subtask lift represents the number of cans that satisfy the condition. \ Subtask insert represents the number of cans that are inserted. \* Subtask unload represents the number of cans that are unloaded.
• Close-Drawer
• Physics Parameters: \ dt: ¹ 240 \ Drawer body collision approximation: Convex Decomposition \ Drawer door collision approximation: SDF Mesh \ Drawer contact offset: 0.01
• Task Definition: \ Final task success if drawer is fully closed \ Subtask reach success if distance between EE and handle on drawer door is smaller than 0.12 \* Subtask move-door success if drawer is closed at least 10% of its joint range
• Open-Drawer
• Physics Parameters: \ dt: ¹ 240 \ Drawer body collision approximation: Convex Decomposition \ Drawer door collision approximation: SDF Mesh \ Drawer contact offset: 0.01
• Task Definition: \ Final task success if drawer is fully open \ Subtask reach success if distance between EE and handle on drawer door is smaller than 0.12 \* Subtask move-door success if drawer is open at least 10% of its joint range
• Stack-Can
• Physics Parameters: \ dt: ¹ 120 \ Can collision approximation: Convex hull \ Can contact offset: 0.0015 \ Plate collision approximation: Convex Decomposition \* Plate contact offset: auto computed
• Task Definition: \ Final task success if horizontal distance (calculated using x,y coordinates) between can and plate is smaller than radius of plate and difference in z values is smaller than 0.02 \ Subtask place success if horizontal distance (calculated using x,y coordinates) between can and plate is smaller than radius of plate and difference in z values is smaller than 0.05
• Stack-Can-Into-Drawer
• Physics Parameters: \ dt: ¹ 240 \ Can collision approximation: Convex hull \* Can contact offset: 0.0015

• \ Plate collision approximation: Convex Decomposition \ Plate contact offset: auto computed \ Drawer body collision approximation: Convex Decomposition \ Drawer door collision approximation: SDF Mesh \ Drawer contact offset: 0.01 \
• Task Definition: \ Final task success if horizontal distance (calculated using x,y coordinates) between can and plate is smaller than radius of plate, difference in z values is smaller than 0.02, and drawer is close. \ Subtask reach drawer success if distance between left EE and drawer handle is smaller than 0.12. \ Subtask reach can success if distance between right EE and can is smaller than 0.12. \ Subtask lift success if height of can increases by at least 0.06.
• Open-Laptop
• Physics Parameters: \ dt: ¹ 120 \ Laptop body collision approximation: Convex Hull \ Laptop lid collision approximation: Convex Hull \ Laptop lid contact offset: auto computed
• Task Definition: \ Final task success if laptop is open by at least 70% of its joint range. \ Subtask move lid success if laptop is open by at least 15% of its joint range.

For robot hands, collision approximation for each link is Convex Decomposition, and contact offset is 0.005.