Abstract
Learning manipulation skills from human videos is promising for scalable robot learning. However, the embodiment mismatch between humans and robots makes this challenging. One promising solution is to learn object-centric actionable affordances that are embodiment-agnostic. In this work, we propose a framework that leverages egocentric human videos with state-of-the-art 3D Structure-from-Motion and hand mesh reconstruction to extract actionable affordances such as visual, grasp, and trajectory affordances that explicitly encode where to interact, how to grasp, and how to move. We construct EgoAffordance, a large-scale dataset comprising 204K episodes with 5.6M visual affordances and 11.6M grasp and trajectory affordances. Building on this, we introduce VLAff, a large vision-language model-based unified foundation model that learns cross-modal correlations across all actionable affordances. Given a visual observation and instruction, VLAff generates visual affordance heatmaps, grasp poses, and trajectories, which are then converted into directly executable actions by utilizing 3D scene information. Through extensive experiments, we demonstrate that VLAff not only achieves state-of-the-art performance on visual affordance prediction, but can also be effectively applied to real robot applications such as zero-shot manipulation and affordance-guided robot learning.
Concept
Actionable Affordance := See+Grasp+Move
See
Visual affordance — a dense heatmap over the image marking where to interact. Its peak is projected to 3D to become the object-centric anchor point.
Grasp
Grasp affordance — the hand configuration describing how to grasp the object at the contact point.
Move
Trajectory affordance — a sequence of post-contact 6D hand poses describing how to move after grasping.
Dataset Pipeline
EgoAffordance turns egocentric human interactions into supervision for where to touch, how to grasp, and how to move.
- Curated episodes
- 204,025
- Visual heatmaps
- 5.78M
- Trajectory sequences
- 11.61M
Sources: EPIC-KITCHENS · Ego4D · Ego-Exo4D · HD-EPIC
Data Generation Pipeline
From each egocentric video segment we extract contact regions, 3D hand poses, and 3D hand trajectories fully automatically. Inpainting removes the human to leave agent-agnostic scenes, and Structure-from-Motion lifts the hand tracks into 3D.
Actionable affordance extraction pipeline from egocentric human video.
-
Find the interaction
Sample a 4-second window at 30 Hz (120 frames) around each annotated action. A vision-language model identifies action boundaries, the object, and the interacting hand.
-
Recover contact and grasp
Hand-object detection, object masks, and fingertip tracking locate contact regions. Hand mesh reconstruction recovers the MANO grasp pose; inpainting removes the human from the scene.
-
Reconstruct motion in 3D
Depth and camera poses lift the hand motion into 3D. Grasp poses and post-contact trajectories are expressed relative to the interaction point for transfer across embodiments.
After filtering incorrect annotations, missing data, and quality issues, 204,025 of the 243,889 extracted episodes remain.
Generated Data Samples
Samples from the EgoAffordance dataset.
Model Architecture
VLAff extends a pretrained vision-language model with three kinds of affordance tokens.
A <SEG> token drives a segmentation decoder that fuses dense features from an
additional vision encoder to produce the visual affordance heatmap. A <GRASP> token
decodes into hand parameters. Trajectories are discretized into a vocabulary of trajectory tokens and
generated autoregressively, so a single VLM captures the correlation between where to touch, how to hold,
and how to move.
VLAff architecture: vision encoder, VLM, and specialized decoders for visual, grasp, and trajectory affordances.
Qualitative Results
Visual affordance prediction on internet data, compared against baselines.
Open drawer
Open fridge
Predicted grasp poses and trajectories lifted into the 3D scene, in a real lab environment.
Robot Deployment
Predicted contact regions, grasp poses, and trajectories guide real-world manipulation. Zero-shot demonstrations are shown below, with playback speed indicated on each clip.
Affordance-Guided RL
BibTeX
@inproceedings{oh2026vlaff,
title = {VLAff: Vision-Language-Affordance Model for Unified Actionable Affordances},
author = {Oh, Jihoon and Kawaharazuka, Kento and Okada, Kei},
booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
year = {2026},
eprint = {2608.05215},
archivePrefix = {arXiv},
primaryClass = {cs.RO}
}