VLAff: Vision-Language-Affordance Model
for Unified Actionable Affordances
Abstract
Learning manipulation skills from human videos is promising for scalable robot learning. However, the embodiment mismatch between humans and robots makes this challenging. One promising solution is to learn object-centric actionable affordances that are embodiment-agnostic. In this work, we propose a framework that leverages egocentric human videos with state-of-the-art 3D Structure-from-Motion and hand mesh reconstruction to extract actionable affordances such as visual, grasp, and trajectory affordances that explicitly encode where to interact, how to grasp, and how to move. We construct EgoAffordance, a large-scale dataset comprising 204K episodes with 5.6M visual affordances and 11.6M grasp and trajectory affordances. Building on this, we introduce VLAff, a large vision-language model-based unified foundation model that learns cross-modal correlations across all actionable affordances. Given a visual observation and instruction, VLAff generates visual affordance heatmaps, grasp poses, and trajectories, which are then converted into directly executable actions by utilizing 3D scene information. Through extensive experiments, we demonstrate that VLAff not only achieves state-of-the-art performance on visual affordance prediction, but can also be effectively applied to real robot applications such as zero-shot manipulation and affordance-guided robot learning.
Concept
Actionable Affordance := See+Grasp+Move
See
Visual affordance — a dense heatmap over the image marking where to interact. Its peak is projected to 3D to become the object-centric anchor point.
Grasp
Grasp affordance — the hand configuration describing how to grasp the object at the contact point.
Move
Trajectory affordance — a sequence of post-contact 6D hand poses describing how to move after grasping.
Dataset Pipeline
Data Generation Pipeline
From each egocentric video segment we extract contact regions, 3D hand poses, and 3D hand trajectories fully automatically. Inpainting removes the human to leave agent-agnostic scenes, and Structure-from-Motion lifts the hand tracks into 3D.
Actionable affordance extraction pipeline from egocentric human video.
Generated Data Samples
Samples from the EgoAffordance dataset.
Model Architecture
VLAff extends a pretrained vision-language model with three kinds of affordance tokens.
A <SEG> token drives a segmentation decoder that fuses dense features from an
additional vision encoder to produce the visual affordance heatmap. A <GRASP> token
decodes into hand parameters. Trajectories are discretized into a vocabulary of trajectory tokens and
generated autoregressively, so a single VLM captures the correlation between where to touch, how to hold,
and how to move.
VLAff architecture: vision encoder, VLM, and specialized decoders for visual, grasp, and trajectory affordances.
Qualitative Results
Visual affordance prediction on internet data, compared against baselines.
Open drawer
Open fridge
Predicted grasp poses and trajectories lifted into the 3D scene, in a real lab environment.
Robot Deployment
BibTeX
@inproceedings{oh2026vlaff,
title = {VLAff: Vision-Language-Affordance Model for Unified Actionable Affordances},
author = {Oh, Jihoon and Kawaharazuka, Kento and Okada, Kei},
booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
year = {2026}
}