Reproducible Multimodal Affordance Prediction

Tommaso Apicella¹ , Alessio Xompero² , and Andrea Cavallaro³

Istituto Italiano di Tecnologia, Italy

2

Independent Researcher

³ EPFL, Switzerland

t.apicella.cs@gmail.com, alessio.xompero@gmail.com, andrea.cavallaro@epfl.ch

Abstract. Affordance prediction is the identification of potential actions an agent can perform on a target object from multimodal inputs. Affordance prediction methods are difficult to evaluate and compare due to heterogeneous problem formulations, inconsistent dataset annotations, incomplete reporting of experimental protocols, and limited information about deployment conditions. These limitations challenge fair benchmarking and performance comparison. To promote transparency, we propose the Affordance Sheet, a documentation detailing task formulation with its input modalities, model architectures and training information, datasets, and experimental protocols. Affordance Sheets enable reproducible benchmarking and reliable evaluation of affordance models for real-world scenarios, including generalisation to novel conditions and human safety.

Keywords: Affordance · Benchmarking · Transparency

Affordances describe the potential actions that an agent can perform on objects in the environment [\[20\]](#page-15-0), inferred from multimodal sensory observations such as visual, depth, acoustic, and tactile data [\[28\]](#page-15-1). Understanding affordances enables the agent to accomplish a task, selecting which objects in the environment to interact with, what actions to perform, and how to execute them. This reasoning within and across modalities, beyond simply perceiving scene and objects, supports applications such as human-robot collaboration [\[48\]](#page-16-0) and wearable robotics [\[5\]](#page-14-0), characterised by unstructured environments and occlusions.

The generic definition of affordance has resulted in different formulations for the prediction of affordances, capturing only a partial aspect of the overall problem (see Fig. [1\)](#page-1-0). Affordance classification [\[41\]](#page-16-1) focuses on identifying plausible actions. Affordance segmentation [\[14\]](#page-14-1) aims to localize relevant interaction regions on objects. Hand pose estimation [\[35\]](#page-16-2) predicts hand configurations that enable the actions. This fragmentation is worsened by the use of datasets focused on specific sensors, with partially overlapped annotation, limited code availability,

A. Xompero did part of the work when he was affiliated with Queen Mary University of London.

<span id="page-1-0"></span>

Fig. 1: Illustrative example of affordance prediction from multimodal inputs (e.g. images, language, point clouds) formulated differently, such as segmentation (top) or hand pose estimation (bottom), and lacking transparent details such as code implementation, model weights, or training and experimental setup. Different reporting makes models not reproducible or comparable, and prevents their deployment in diverse real-world environments and in human-centred scenarios (e.g. human-robot interaction).

inconsistent training setups, and partially reported evaluation protocols. Moreover, the design of methods and learning-based models is mostly done in controlled settings with objects placed fully visible in laboratory-like scenes (e.g. a tabletop) and considering a limited number of object types [\[3,](#page-14-2) [23,](#page-15-2) [29,](#page-15-3) [31,](#page-15-4) [40,](#page-16-3) [43\]](#page-16-4). Multimodal affordance prediction models should be designed and evaluated to assess their generalization to diverse objects and real-world scenarios (including interactions with people manipulating the target object), their robustness to occlusions from other objects or from the hand of a person holding the object, and safety for the person [\[44\]](#page-16-5). All the above factors result in misleading and incomplete insights, slowing progress in the field and limiting the benchmarking of models for reliable real-world deployment.

Reproducibility, defined as obtaining the same results given the same conditions (i.e. data, training and testing setups, and trained model) [\[47\]](#page-16-6), is a central concern in machine learning [\[47\]](#page-16-6) with works highlighting inconsistencies in experimental practices and proposing standardized reporting tools [\[19,](#page-15-5) [38\]](#page-16-7). Model cards [\[38\]](#page-16-7) and datasheets [\[19\]](#page-15-5) aim to improve transparency when releasing machine learning models and datasets, respectively. Model cards [\[38\]](#page-16-7) help the identification of model limitations by describing the method, the experimental setup, and the applications or conditions leading to underperformance. Datasheets for datasets [\[19\]](#page-15-5) facilitate the description of data limitations by detailing dataset collection and annotation procedures, intended use, and potential biases. These reporting practices have been used to document multimodal models such as Vision-Language-Action Models (VLAs) [\[25,](#page-15-6)[26\]](#page-15-7) or World Action Models [\[18,](#page-15-8)[37\]](#page-16-8) and datasets, providing details and intended use about methods and data. However, these data- and model-centric tools do not capture the task-specific ambiguities that characterize affordance prediction, such as the diversity of problem

<span id="page-2-0"></span>

Fig. 2: Examples of applications benefiting from affordance prediction: immersive tutorial [\[21\]](#page-15-9) (left), human-robot collaboration [\[48\]](#page-16-0) (centre), and human-to-human collaboration with wearable robotics [\[5\]](#page-14-0) (right).

formulations, the definition and semantics of affordances, and the characteristics of agent–object interactions. As a result, critical aspects required for fair comparison, such as how affordances are defined and how predictions translate into actionable behaviours, are not specified. Additionally, model cards and datasheet do not explicitly account for the validation of methods through in-the-wild conditions or through deployment (e.g. on a real or simulated robot) to assess the generalisation and robustness of the performance. Reproducibility in affordance prediction is critical because learned models can directly perform physical interactions with the environment. Non-reproducible results compromise robustness to occlusions, safety, and generalization to diverse real-world scenarios. Ensuring reproducibility therefore supports both the credibility of research findings and the development of reliable systems capable of consistent and safe interaction with humans in real-world environments (see Fig. [2\)](#page-2-0).

In this work, we surface reproducibility issues of current affordance works, which lead to inconsistent comparisons across methods. To address these challenges, we introduce the Affordance Sheet, an Open Science document inspired by Model Cards [\[38\]](#page-16-7), detailing the tackled task, the dataset, the method, and the experimental setup and validation. Additionally, we provide and discuss four Affordance Sheet of state-of-the-art models for affordance prediction. Affordance Sheet enables to catalogue solutions designed to work in unconstrained settings, compare characteristics and validations across methods, and improve reproducibility and promote transparency[¹](#page-2-1) . The tool is generic and can be extended to methods such as VLAs and affordance foundation models, and to tasks beyond affordance prediction.

2 Reproducibility Challenges

Reproducibility challenges (RCs) of affordance prediction (see Fig. [3\)](#page-3-0) include:

• 1. data availability for benchmarking (RC1);
• 2. availability of a method's implementation (RC2);

<span id="page-2-1"></span>¹ Project webpage at https://apicis.github.io/aff-sheet.

<span id="page-3-0"></span>

Fig. 3: Visualization of the main challenges tackled by Affordance Sheet sections. KEYS – RC: reproducibility challenge, RoV: robot validation

• 3. availability of trained models (RC3);
• 4. details of experimental setups (RC4); and
• 5. details of performance measures for evaluation (RC5).

2.1 Datasets and Benchmarks

Table [1](#page-4-0) reports the main characteristics of the available datasets and benchmarks in affordance prediction. None of the datasets is collected for benchmarking methods under different in-the-wild conditions (RC1), such as illumination, clutter, or hand-occlusions. Each affordance dataset is collected for a specific formulation, and not re-used across different affordance tasks, limiting fair comparison and preventing comprehensive benchmarking. One of the main obstacles to obtain large-scale data collections to assess models performance is the annotation. Precise annotations are manual and time-consuming, limiting the size of most datasets to less than 50,000 images and less than 10 affordance categories. For this reason, the largest datasets, such as HANDAL [\[23\]](#page-15-2) or CHOC-AFF [\[3\]](#page-14-2), either used synthetic data to scale the annotation or adapted methods to provide weakly or self-supervised annotation.

Most of the affordance datasets [\[4,](#page-14-3) [7,](#page-14-4) [13,](#page-14-5) [30–](#page-15-10)[32,](#page-15-11) [40,](#page-16-3) [55\]](#page-17-0) are acquired in controlled laboratory environments. A single unoccluded object is placed on a planar surface (e.g. a tabletop), with fixed camera viewpoints and illumination, while varying only the object category or instance. For instance, both UMD [\[40\]](#page-16-3) and Multi-View [\[31\]](#page-15-4) include multiple object categories collected under pre-defined acquisition settings, such as constant lighting and rotating platforms. These controlled conditions simplify data collection and evaluation but do not adequately capture the variability encountered during real-world deployment, limiting the generalization of trained models to cluttered scenes, diverse backgrounds, or illumination variations. Other datasets [\[3,](#page-14-2) [8,](#page-14-6) [23,](#page-15-2) [36,](#page-16-9) [43,](#page-16-4) [49\]](#page-16-10) included images of objects with occlusions caused by surrounding objects or human hands interacting with the scene to increase realism. Such conditions substantially increase the difficulty of affordance prediction while better reflecting practical robotic

<span id="page-4-0"></span>Table 1: Characteristics of datasets for affordance prediction grouped by formulations.

KEYS – VIS: vision with RGB(-D), LAN: language, PCL: point cloud, # Images: number of images, OBJ: number of object categories, AFF: number of affordance categories, Tran.: transparency, 3PV: third person view, HOc: hand-occlusion; AFFC: affordance classification; AFFG: affordance grounding; HPE: hand pose estimation; GDET: grasping detection; AFFDS: affordance detection and segmentation; : considered, : not considered, G#: partly considered.

applications. Hand-induced occlusions are common in human-robot collaboration scenarios, where inaccurate affordance estimation may result in unsafe or unintended interactions, posing risks to human operators [\[3,](#page-14-2) [45\]](#page-16-14).

Most of previous works [\[14,](#page-14-1)[22,](#page-15-15)[42,](#page-16-15)[56](#page-17-6)[–58\]](#page-17-7) trained models on the training split of a specific dataset and then compared the models performance on the testing split of one or more datasets. Cross-dataset evaluations are mostly avoided due to partial overlapping of affordance classes or of object categories, across the selected datasets [\[2\]](#page-14-14). Datasets such as UMD [\[40\]](#page-16-3), IIT-AFF [\[43\]](#page-16-4), and Multi-View [\[31\]](#page-15-4) share some of the object and affordance classes but labelled with different conventions, making the comparison of models trained on different datasets difficult. As a consequence, researchers train multiple versions of the same model, adapted to the classes of a specific dataset. Additional documentation, such as metadata, help researchers train or evaluate methods only on common categories by re-ordering them. Moreover, relying only on a single benchmark can lead to limited and not generalisable considerations on model rankings. For example, images in UMD and Multi-View are collected in a laboratory environment with static conditions, e.g. a fixed camera oriented towards a table where an object is placed (camera-object distance is almost always the same) [\[31,](#page-15-4) [40\]](#page-16-3). However, in real scenarios the camera might be closer or farther from objects compared to the training setting, hence the performance on the benchmark might not reflect in-the-wild performance.

2.2 Methods and Experimental Setups

The lack of publicly available implementation of methods (RC2) [\[22,](#page-15-15)[56–](#page-17-6)[58\]](#page-17-7), the lack of publicly available trained models (RC3) [\[22,](#page-15-15) [42,](#page-16-15) [56](#page-17-6)[–58\]](#page-17-7), and the lack of details of experimental setups (RC4) [\[14,](#page-14-1)[22,](#page-15-15)[42,](#page-16-15)[56](#page-17-6)[–58\]](#page-17-7) can challenge researchers in reproducing previous works for comparative evaluations. The release of the model trained weights and the implementation of the method and inference pipeline is a crucial aspect for reproducibility, especially for deep-learning based models, allowing other researchers to test models on their own data without re-training. The availability of model implementation and weights is important when researchers need a comparison, as re-training the model can be too time- and resource-consuming. In case a new dataset is proposed and a previous method needs re-training, only the method implementation is sufficient. The reimplementation of methods and setup is time-consuming and prone to errors, and not always leads to the expected outcome (i.e. results are not replicable or findings are not reproducible). To avoid this issue and save time, researchers report results from previous works [\[22,](#page-15-15) [56](#page-17-6)[–58\]](#page-17-7), resulting in misleading findings and conclusions when different experimental conditions are used.

Using the same experimental setup to train and test affordance models allows a fair comparison and validation of new technical contributions. When releasing the training and testing code is not possible, reporting all details to reproduce a setup becomes fundamental. The experimental setup details include training hyper-parameter values, chosen data splits, image pre-processing (normalisation and cropping procedures), and post-processing. The lack of experimental setup details causes methods for affordance detection and segmentation to be not reproducible [\[14,](#page-14-1) [22,](#page-15-15) [42,](#page-16-15) [56–](#page-17-6)[58\]](#page-17-7). For example, AffordanceNet and BPN did not include image resize during training and testing phases [\[14,](#page-14-1)[56\]](#page-17-6), whereas DRNAtt, RANet, and GSE did not include these details for the testing phase. Gu et al.'s work [\[22\]](#page-15-15) omitted the parameters of the optimizers used during training. Apicella et al.'s work [\[2\]](#page-14-14) showed that the lack of details in the experimental setup led to unfair comparisons.

Table [2](#page-6-0) reports the training and testing setups of affordance detection and segmentation methods on the UMD dataset. Despite being trained and tested on the same dataset, models' performance is not directly comparable due to inconsistencies in the setups such as the image resize procedure and augmentation procedure during training. Inconsistencies can also be present in previous methods adapted into a baseline to compare with. For example, AffordanceDiffusion [\[55\]](#page-17-0) is compared with the coarse hand prediction of GanHand [\[8\]](#page-14-6) due to

<span id="page-6-0"></span>Table 2: Comparison of training/testing setups used by different methods for affordance detection and segmentation on the UMD dataset [\[40\]](#page-16-3). Due to the setup inconsistencies, direct comparison among models performance is unfair.

KEYS – •: considered, ◦: not considered, FLIP: flipping, SCALE: scaling, ROT: rotating, JIT: colour jittering, Train.: training set, Test.: testing set

the missing annotation of the object pose in the dataset. However, since a part of the architecture and of the training procedure is missing, the result is only a proxy to the (unknown) performance of GanHand.

The redefinition of the affordance problem can also result in experimental validations that ignore datasets and benchmarks of partially overlapping formulations. For example, works on affordance grounding [\[36\]](#page-16-9) do not compare the performance of proposed methods with that of affordance segmentation methods [\[22,](#page-15-15) [42,](#page-16-15) [57,](#page-17-8) [58\]](#page-17-7), despite the similar problem formulation [\[36,](#page-16-9) [42\]](#page-16-15). Methods for affordance segmentation output a binary mask for each action in a predefined set of classes, whereas methods for affordance grounding output a confidence map describing where an action known a priori can take place in the image. Despite these differences, comparing methods for both affordance grounding and affordance detection and segmentation can explain if using action as input (affordance grounding) to a model provides any advantage.

2.3 Performance Measures

The performance of different methods for each affordance formulation [\[14,](#page-14-1)[35,](#page-16-2)[41\]](#page-16-1) was evaluated using scores or metrics to quantify the discrepancy between predictions and annotations. Describing a performance measure help other researchers understand if the experiment validates their claim or if a different measure should be chosen. Providing the mathematical formulation of the measures helps disambiguate different implementations of the same score, especially when a public evaluation toolkit is not used or referred to. For example, mean IoU can be the average of all the IoU s between prediction and annotation, or the IoU considering the full set of predictions and annotations. Previous works evaluated a few methods with different performance measures or datasets, making comparison and ranking not possible. For example, the performance of STRAP [\[9\]](#page-14-15) was measured using IoU instead of F w β , as most of other methods do [\[14,](#page-14-1) [22,](#page-15-15) [42,](#page-16-15) [56–](#page-17-6)[58\]](#page-17-7).

To support reproducibility in affordance prediction, we propose the Affordance Sheet, an organised collection of good practices favouring fair comparisons. Each section of the Affordance Sheet (see Table [3\)](#page-8-0) addresses a specific reproducibility challenge providing an informative tool to catalogue and compare methods.

The first section identifies which problems the affordance model tackles, the inputs used by the model, and the presence of the human in the scenario. This section helps researchers understand what are the competing methods and assess their performance of solutions under the same inputs and conditions. When proposing a new problem partially overlapping with another one, previous models can be used or adapted to validate the method. For example, selecting the channel of an affordance segmentation output based on the action considered by the affordance grounding method enables the comparison between methods for affordance segmentation and methods for affordance grounding. To compare the grounding and segmentation outputs, the grounding confidence map can be converted to a binary mask via thresholding. Alternatively, the segmentation map can be converted to a confidence map by using Gaussian blur [\[36\]](#page-16-9). In the sheet, we unify affordance detection and segmentation with affordance grounding under functional segmentation; we consider affordance classification as functional classification, and grasping detection as a case of hand pose estimation.

The second section describes the datasets (RC1) used by the proposed solution to detail their characteristics and share the link to the data and the license informing about data permissions. Future benchmarks should release a detailed description on how to use and visualize data so that researchers can get acquainted with the format, and should evaluate models under different conditions, such as generalization across unseen object instances, novel object categories, camera viewpoints, lighting conditions, clutter, partial occlusions, and different sensing modalities. Benchmarks for different tasks, such as COCO for object detection and instance segmentation [\[34\]](#page-15-16), release only the training and validation sets, while keeping a private testing set not to bias the architecture designer. On the contrary, benchmarks for affordance prediction, such as UMD and IIT-AFF, release both the training and testing data, posing the risk of modifying the methods to improve performance scores on the testing set rather than formulating contributions that advance the field.

The third section highlights the model characteristics (RC2, RC3). Providing model cards [\[38\]](#page-16-7), along with its implementation and trained weights, helps detail the description of models, supporting other researchers to build upon. When not available, we encourage the re-implementation and retraining of the models as a contribution for the community. Previous works [\[2,](#page-14-14)[3\]](#page-14-2) re-implemented, retrained, and released models for affordance detection and segmentation due to the lack of publicly available models. We also recommend providing a link to the trained model's weights and a license detailing the allowed uses[²](#page-7-0) .

<span id="page-7-0"></span>² Publicly-available software without a license is automatically protected by copyright.

Other researchers cannot use the method implementation to reproduce results.

<span id="page-8-0"></span>Table 3: Affordance Sheet, inspired by Model Cards [\[38\]](#page-16-7), to favour transparency and reproducibility of visual affordance prediction methods.

The fourth section provides the details of the experimental setup to train and evaluate methods under the same conditions (RC4), including pre-processing and post-processing information such as data splits, resize procedures, data normalisation, and hyper-parameters choice. The lack of these details can result in models with significantly different weights and in unfair comparisons.

The fifth section focuses on the criteria to validate and compare methods (RC5). Providing a stand-alone toolkit implementing the performance measures ensures the replicability of the results across different works. For affordance prediction, we recommend evaluating the performance of models using more than one measure to provide a more comprehensive analysis while identifying different aspects and limitations of the models. For example, in affordance segmentation, precision focuses on how many of the predicted pixels have the correct class and recall emphasizes how many of the annotated pixels are correctly predicted. Therefore, computing more than one score (and avoiding using a single score aggregating multiple performance measures) reduces the risk of drawing misleading conclusions that are based only on partial results.

The last section describes the potential validation of the method in terms of generalisation, robustness, safety, and robotic deployment. The Affordance Sheet documents the assessment of the model generalization across objects and environments unseen during training, robustness to realistic sensing disturbances such as occlusions and illumination changes, and safety through standardized tests and protocols for interaction with humans. In previous works, few of the methods were validated using a real robotic platform [\[14,](#page-14-1) [43,](#page-16-4) [56\]](#page-17-6). When a robot experiment can be performed, we recommend reporting the characteristics of the setup, the robotic hand specifics, and the description of the experiment in terms of object, conditions, and physics engine in case of simulated robot. This transparent reporting allows researchers to assess methods using a common platform. All these aspects promote more transparent assessment of multimodal affordance models in real-world, human-centred scenarios.

4 Comparison of Previous Methods

We provide examples of Affordance Sheets (empty fields are not reported): Ego-Topo [\[41\]](#page-16-1) for functional classification in Table [4,](#page-10-0) ACANet [\[3\]](#page-14-2) for affordance segmentation in Table [5,](#page-11-0) Multi-FinGAN [\[35\]](#page-16-2) and HUG [\[53\]](#page-17-5) for hand pose estimation in Table [6](#page-12-0) and Table [7,](#page-13-1) respectively. These examples highlight substantial differences in the practices adopted by the methods, revealing reproducibility gaps difficult to identify from the original publications alone.

EgoTopo, MultiFin-Gan, and ACANet provide access to the training data while the details about metadata such as dataset versions, licenses, and persistent identifiers is inconsistent, complicating long-term reproducibility and dataset reuse. The Affordance Sheets also reveal notable differences in the reporting of experimental protocols. ACANet provides the most complete description, including training, validation and multiple test splits, extensive hyper-parameters, and image preprocessing details. In contrast, EgoTopo omits the training and validation split sizes and several preprocessing parameters, while Multi-FinGAN does not report validation or testing partitions and leaves several training hyperparameters unspecified. Such inconsistencies make it difficult to reproduce published results or determine whether observed performance differences arise from methodological innovations or from variations in the experimental setup. Similarly, the reported evaluation protocols vary substantially. ACANet and HUG provide a formal definition of the Jaccard Index and Fingertip contact error, whereas EgoTopo and Multi-FinGAN only describe their evaluation metrics without specifying their mathematical formulation. No method described the limitations of the chosen performance measures. Explicitly documenting the assumptions and limitations of performance measures is important because dif-

Table 4: Affordance Sheet for EgoTopo [\[41\]](#page-16-1).

<span id="page-10-0"></span>

ferent metrics capture different performance aspects, favouring different model behaviours. Neither EgoTopo nor ACANet describe any robotic deployment, leaving open the question of how their predictions transfer to real manipulation scenarios. In contrast, Multi-FinGAN and HUG include a real-world evaluation by specifying the robot platform, end-effector, sensing setup, execution protocol, and success criterion. HUG's generalisation to different objects shape is assessed using 90 object categories unseen during training with different sizes and geometries. The generalisation of ACANet and Multi-FinGAN is tested on real images, different from those in the training data (synthetic or mixed-reality images). None of the methods assessed robustness or human safety.

Table 5: Affordance sheet for ACANet [\[3\]](#page-14-2).

<span id="page-11-0"></span>

5 Conclusion

In this work, we identified the factors that undermine fair benchmarking, reproducible evaluation, and deployment-oriented assessment of multimodal models for affordance prediction: the diversity of problem formulations, the limited availability of implementations, and the insufficient reporting of experimental setups. To promote transparency, we introduced the Affordance Sheet, a tool designed to systematically document models, datasets, and validation protocols, focusing on generalisation to diverse objects and environments, robustness to occlusions, and scenarios where the agent interacts with humans (human safety). Affordance

Table 6: Affordance Sheet for Multi-FinGAN [\[35\]](#page-16-2).

<span id="page-12-0"></span>

Sheets aim to facilitate comparable research outcomes beyond quantitative performance and expose missing information that directly affects reproducibility and fair benchmarking. Affordance Sheet represents the first example of reporting practice for multimodal affordance prediction models, and supports a more informed design of the next-generation of learning-based robotic systems operating reliably in diverse real-world environments. The tool can also be generalised

Table 7: Affordance Sheet for HUG [\[53\]](#page-17-5).

<span id="page-13-1"></span>

to tasks different from affordance prediction and that would require similar transparent documentation.

References

• <span id="page-13-0"></span>1. Ainetter, S., Fraundorfer, F.: End-to-end trainable deep neural network for robotic grasp detection and semantic segmentation from RGB. In: Int. Conf. Robot. Au-

• tom. (2021). https://doi.org/10.1109/ICRA48506.2021.9561398
• <span id="page-14-14"></span>2. Apicella, T., Xompero, A., Gastaldo, P., Cavallaro, A.: Segmenting object affordances: Reproducibility and sensitivity to scale. In: Eur. Conf. Comput. Vis. Workshops (2024). https://doi.org/10.1007/978-3-031-92591-7\\_18
• <span id="page-14-2"></span>3. Apicella, T., Xompero, A., Ragusa, E., Berta, R., Cavallaro, A., Gastaldo, P.: Affordance segmentation of hand-occluded containers from exocentric images. In: Int. Conf. Comput. Vis. Workshops (2023). https://doi.org/10.1109/ICCVW60793. 2023.00204
• <span id="page-14-3"></span>4. Asif, U., Tang, J., Harrer, S.: GraspNet: An Efficient Convolutional Neural Network for Real-time Grasp Detection for Low-powered Devices. In: Int. Joint Conf. Artificial Intell. (2018). https://doi.org/10.24963/ijcai.2018/677
• <span id="page-14-0"></span>5. Castro, M.N., Dosen, S.: Continuous semi-autonomous prosthesis control using a depth sensor on the hand. Frontiers in Neurorobotics 16 (2022). https://doi. org/10.3389/fnbot.2022.814973
• <span id="page-14-7"></span>6. Christensen, A.D., Lehotsk`y, D., Jørgensen, M.W., Chrysostomou, D.: Learning to segment object affordances on synthetic data for task-oriented robotic handovers. In: Brit. Mach. Vis. Conf. (2022), https://pdf-files.bmvc2022.org/0544.pdf
• <span id="page-14-4"></span>7. Chu, F., Xu, R., Vela, P.: Learning affordance segmentation for real-world robotic manipulation via synthetic images. IEEE Robot. Autom. Lett. 4(2) (2019). https: //doi.org/10.1109/LRA.2019.2894439
• <span id="page-14-6"></span>8. Corona, E., Pumarola, A., Alenya, G., Moreno-Noguer, F., Rogez, G.: Ganhand: Predicting human grasp affordances in multi-object scenes. In: Conf. Comput. Vis. Pattern Recognit. (2020). https://doi.org/10.1109/CVPR42600.2020.00508
• <span id="page-14-15"></span>9. Cui, L., Chen, X., Zhao, H., Zhou, G., Zhu, Y.: STRAP: Structured object affordance segmentation with point supervision (2023). https://doi.org/10.48550/ arXiv.2304.08492
• <span id="page-14-12"></span>10. Damen, D., et al.: Scaling egocentric vision: The EPIC-KITCHENS dataset. In: Eur. Conf. Comput. Vis. (2018). https://doi.org/10.1007/978-3-030-01225- 0\\_44
• <span id="page-14-9"></span>11. Delitzas, A., Takmaz, A., Tombari, F., Sumner, R., Pollefeys, M., Engelmann, F.: Scenefun3D: Fine-grained functionality and affordance understanding in 3D scenes. In: Conf. Comput. Vis. Pattern Recognit. (2024). https://doi.org/10. 1109/CVPR52733.2024.01377
• <span id="page-14-8"></span>12. Deng, S., Xu, X., Wu, C., Chen, K., Jia, K.: 3D AffordanceNet: A benchmark for visual object affordance understanding. In: Conf. Comput. Vis. Pattern Recognit. (2021). https://doi.org/10.1109/CVPR46437.2021.00182
• <span id="page-14-5"></span>13. Depierre, A., Dellandréa, E., Chen, L.: Jacquard: A large scale dataset for robotic grasp detection. In: Int. Conf. Intell. Robot Syst. (2018). https://doi.org/10. 1109/IROS.2018.8593950
• <span id="page-14-1"></span>14. Do, T., Nguyen, A., Reid, I.: AffordanceNet: An end-to-end deep learning approach for object affordance detection. In: Int. Conf. Robot. Autom. (2018). https://doi. org/10.1109/ICRA.2018.8460902
• <span id="page-14-13"></span>15. Fang, H.S., Wang, C., Gou, M., Lu, C.: GraspNet-1Billion: A large-scale benchmark for general object grasping. In: Conf. Comput. Vis. Pattern Recognit. (2020). https://doi.org/10.1109/CVPR42600.2020.01146
• <span id="page-14-10"></span>16. Fang, K., Wu, T.L., Yang, D., Savarese, S., Lim, J.J.: Demo2vec: Reasoning object affordances from online videos. In: Conf. Comput. Vis. Pattern Recognit. (2018). https://doi.org/10.1109/CVPR.2018.00228
• <span id="page-14-11"></span>17. Gao, X., Zhang, P., Qu, D., Wang, D., Wang, Z., Ding, Y., Zhao, B.: Learning 2D invariant affordance knowledge for 3D affordance grounding. In: AAAI Conf. Artif. Intell. (2025). https://doi.org/10.1609/aaai.v39i3.32318

• <span id="page-15-8"></span>18. GEAR-Dreams: Dreamzero-droid. https : / / huggingface . co / GEAR - Dreams / DreamZero-DROID (2025)
• <span id="page-15-5"></span>19. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J.W., Wallach, H., Iii, H.D., Crawford, K.: Datasheets for datasets. Communications of the ACM 64(12), 86–92 (2021). https://doi.org/10.1145/3458723
• <span id="page-15-0"></span>20. Gibson, J.J., Carmichael, L.: The senses considered as perceptual systems. Houghton Mifflin Boston 2(1) (1966)
• <span id="page-15-9"></span>21. Grauman, K., Westbury, A., Torresani, L., Kitani, K., Malik, J., et al.: Ego-Exo4D: Understanding skilled human activity from first-and third-person perspectives. In: Conf. Comput. Vis. Pattern Recognit. (2024). https://doi.org/10.1109/ CVPR52733.2024.01834
• <span id="page-15-15"></span>22. Gu, Q., Su, J., Yuan, L.: Visual affordance detection using an efficient attention convolutional neural network. Neurocomputing 440 (2021). https://doi.org/10. 1016/j.neucom.2021.01.018
• <span id="page-15-2"></span>23. Guo, A., et al.: HANDAL: A dataset of real-world manipulable object categories with pose annotations, affordances, and reconstructions. In: Int. Conf. Intell. Robot Syst. (2023). https://doi.org/10.1109/IROS55552.2023.10341672
• <span id="page-15-12"></span>24. Hedvig, K., Javier, R., Danica, K.: Visual object-action recognition: Inferring object affordances from human demonstration. Comput. Vis. Image Understanding 115(1) (2011). https://doi.org/10.1016/j.cviu.2010.08.002
• <span id="page-15-6"></span>25. Hugging Face LeRobot Team: π0.5 (pi05) base model card. https://huggingface. co/lerobot/pi05\\_base (2025)
• <span id="page-15-7"></span>26. Hugging Face LeRobot Team: Smolvla base model card. https://huggingface. co/lerobot/smolvla\\_base (2025)
• <span id="page-15-13"></span>27. Hussain.S, S., Liu, L., Xu, W., Lu, C.: FPHA-Afford: A domain-specific benchmark dataset for occluded object affordance estimation in human-object-robot interaction. In: Int. Conf. Image Process. (2020). https://doi.org/10.1109/ICIP40778. 2020.9190733
• <span id="page-15-1"></span>28. Jamone, L., Ugur, E., Cangelosi, A., Fadiga, L., Bernardino, A., Piater, J., Santos-Victor, J.: Affordances in psychology, neuroscience, and robotics: A survey. IEEE Trans. Cogn. Dev. Syst. 10(1) (2016). https://doi.org/10.1109/TCDS.2016. 2594134
• <span id="page-15-3"></span>29. Jiang, J., Cao, G., Do, T., Luo, S.: A4t: Hierarchical affordance detection for transparent objects depth reconstruction and manipulation. IEEE Robot. Autom. Lett. 7(4) (2022). https://doi.org/10.1109/LRA.2022.3191231
• <span id="page-15-10"></span>30. Jiang, Y., Moseson, S., Saxena, A.: Efficient grasping from RGBD images: Learning using a new rectangle representation. In: Int. Conf. Robot. Autom. (2011). https: //doi.org/10.1109/ICRA.2011.5980145
• <span id="page-15-4"></span>31. Khalifa, Z.O., Shah, S.A.A.: A large scale multi-view rgbd visual affordance learning dataset. In: Int. Conf. Image Process. (2023). https://doi.org/10.1109/ ICIP49359.2023.10222906
• <span id="page-15-11"></span>32. Lakani, S.R., Rodríguez-Sánchez, A.J., Piater, J.: Towards Affordance Detection for Robot Manipulation using Affordance for Parts and Parts for Affordance. Auton. Robots 43(5) (2019). https://doi.org/10.1007/s10514-018-9787-5
• <span id="page-15-14"></span>33. Li, Y., Zhao, N., Xiao, J., Feng, C., Wang, X., Chua, T.: LASO: Language-guided affordance segmentation on 3D object. In: Conf. Comput. Vis. Pattern Recognit. (2024). https://doi.org/10.1109/CVPR52733.2024.01351
• <span id="page-15-16"></span>34. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: Common Objects in Context. In: Eur. Conf. Comput. Vis. (2014). https://doi.org/10.1007/978-3-319-10602-1\\_48

• <span id="page-16-2"></span>35. Lundell, J., et al.: Multi-FinGAN: Generative coarse-to-fine sampling of multifinger grasps. In: Int. Conf. Robot. Autom. (2021). https://doi.org/10.1109/ ICRA48506.2021.9561228
• <span id="page-16-9"></span>36. Luo, H., Zhai, W., Zhang, J., Cao, Y., Tao, D.: Learning affordance grounding from exocentric images. In: Conf. Comput. Vis. Pattern Recognit. (2022). https: //doi.org/10.1109/CVPR52688.2022.00229
• <span id="page-16-8"></span>37. Microsoft Research: WHAM: World and Human Action Model. https : / / huggingface.co/microsoft/wham (2025)
• <span id="page-16-7"></span>38. Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I.D., Gebru, T.: Model cards for model reporting. In: Conf. Fairness, Accountability, Transparency (2019). https://doi.org/10.1145/3287560. 3287596
• <span id="page-16-12"></span>39. Mur-Labadia, L., Guerrero, J.J., Martinez-Cantin, R.: Multi-label affordance mapping from egocentric vision. In: Int. Conf. Comput. Vis. (2023). https://doi.org/ 10.1109/ICCV51070.2023.00483
• <span id="page-16-3"></span>40. Myers, A., Teo, C.L., Fermüller, C., Aloimonos, Y.: Affordance detection of tool parts from geometric features. In: Int. Conf. Robot. Autom. (2015). https://doi. org/10.1109/ICRA.2015.7139369
• <span id="page-16-1"></span>41. Nagarajan, T., Li, Y., Feichtenhofer, C., Grauman, K.: Ego-topo: Environment affordances from egocentric video. In: Conf. Comput. Vis. Pattern Recognit. (2020). https://doi.org/10.1109/CVPR42600.2020.00024
• <span id="page-16-15"></span>42. Nguyen, A., Kanoulas, D., Caldwell, D., Tsagarakis, N.: Detecting object affordances with convolutional neural networks. In: Int. Conf. Intell. Robot Syst. (2016). https://doi.org/10.1109/IROS.2016.7759429
• <span id="page-16-4"></span>43. Nguyen, A., Kanoulas, D., Caldwell, D., Tsagarakis, N.: Object-based affordances detection with convolutional neural networks and dense conditional random fields. In: Int. Conf. Intell. Robot Syst. (2017). https://doi.org/10.1109/IROS.2017. 8206484
• <span id="page-16-5"></span>44. Pang, Y.L., Xompero, A., Oh, C., Cavallaro, A.: Towards safe human-to-robot handovers of unknown containers. In: IEEE Int. Conf. Robot and Human Interactive Comm. (2021). https://doi.org/10.1109/RO-MAN50785.2021.9515350
• <span id="page-16-14"></span>45. Pang, Y.L., Xompero, A., Oh, C., Cavallaro, A.: Stereo hand-object reconstruction for human-to-robot handover. IEEE Robot. Autom. Lett. 10(6) (2025). https: //doi.org/10.1109/LRA.2025.3562790
• <span id="page-16-11"></span>46. Pieropan, A., Ek, C.H., Kjellström, H.: Functional object descriptors for human activity modeling. In: Int. Conf. Robot. Autom. (2013). https://doi.org/10. 1109/ICRA.2013.6630736
• <span id="page-16-6"></span>47. Pineau, J., Vincent-Lamarre, P., Sinha, K., Larivière, V., Beygelzimer, A., d'Alché Buc, F., Fox, E., Larochelle, H.: Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program). J. Mach. Learn. Res. 22(164) (2021), https://jmlr.org/papers/volume22/20-303/20-303.pdf
• <span id="page-16-0"></span>48. Sanchez-Matilla, R., et al.: Benchmark for human-to-robot handovers of unseen containers with unknown filling. IEEE Robot. Autom. Lett. 5(2) (2020). https: //doi.org/10.1109/LRA.2020.2969200
• <span id="page-16-10"></span>49. Sawatzky, J., Srikantha, A., Gall, J.: Weakly supervised affordance detection. In: Conf. Comput. Vis. Pattern Recognit. (2017). https://doi.org/10.1109/CVPR. 2017.552
• <span id="page-16-13"></span>50. Suchi, M., Patten, T., Fischinger, D., Vincze, M.: EasyLabel: A semi-automatic pixel-wise object annotation tool for creating robotic RGB-D datasets. In: Int. Conf. Robot. Autom. (2019). https://doi.org/10.1109/ICRA.2019.8793917

• <span id="page-17-2"></span>51. Sun, J., Moore, J.L., Bobick, A., Rehg, J.M.: Learning visual object categories for robot affordance prediction. Int. J. Robot. Res. 29(2-3) (2010). https://doi.org/ 10.1177/0278364909356602
• <span id="page-17-4"></span>52. Wang, R., Zhang, J., Chen, J., Xu, Y., Li, P., Liu, T., Wang, H.: Dexgraspnet: A large-scale robotic dexterous grasp dataset for general objects based on simulation. In: Int. Conf. Robot. Autom. (2023). https://doi.org/10.1109/ICRA48891.2023. 10160982
• <span id="page-17-5"></span>53. Wu, K.Y., Zhou, T., Tu, I., Yan, B., Guzey, I., Fouhey, D., Shan, D., Pinto, L.: Human universal grasping. In: arXiv:2606.17054 [cs.RO] (2026). https://doi. org/10.48550/arXiv.2606.17054
• <span id="page-17-3"></span>54. Yang, Y., Zhai, W., Luo, H., Cao, Y., Luo, J., Zha, Z.J.: Grounding 3D object affordance from 2D interactions in images. In: Int. Conf. Comput. Vis. (2023). https://doi.org/10.1109/ICCV51070.2023.01001
• <span id="page-17-0"></span>55. Ye, Y., et al.: Affordance diffusion: Synthesizing hand-object interactions. In: Conf. Comput. Vis. Pattern Recognit. (2023). https://doi.org/10.1109/CVPR52729. 2023.02153
• <span id="page-17-6"></span>56. Yin, C., Zhang, Q.: Object affordance detection with boundary-preserving network for robotic manipulation tasks. Neural. Comput. Appl. 34(20) (2022). https:// doi.org/10.1007/s00521-022-07446-4
• <span id="page-17-8"></span>57. Zhang, Y., Li, H., Ren, T., Dou, Y., Li, Q.: Multi-scale fusion and global semantic encoding for affordance detection. In: Int. Joint Conf. Neural Netw. (2022). https: //doi.org/10.1109/IJCNN55064.2022.9892363
• <span id="page-17-7"></span>58. Zhao, X., Cao, Y., Kang, Y.: Object affordance detection with relationship-aware network. Neural. Comput. Appl. 32(18) (2020). https://doi.org/10.1007/ s00521-019-04336-0
• <span id="page-17-1"></span>59. Zheng, X., Zeng, Z., Zhang, J.: High-level object affordance recognition. In: Int. Conf. Inf. Autom. (2018). https://doi.org/10.1109/ICInfA.2018.8812515