VERAGMIL: Virtual Environment for Scooping Granular Foods with Imitation Learning Models

Amanuel Ergogo1,³ , Diego Dall'Alba1,² and Przemyslaw Korzeniowski¹


Abstract— Robot-Assisted Feeding (RAF) systems are essential for assisting individuals with disabilities or motor impairments in eating tasks. Manipulating granular food items, such as rice and beans, poses significant challenges due to their dynamic physical properties. Learning from human demonstrations offers a promising solution, but acquiring high-quality demonstrations is complex. To address this, we present VERAGMIL, a framework that combines a high-fidelity simulator with an intuitive Virtual Reality (VR) interface for recording demonstrations and supporting different imitation learning methods. VERAGMIL provides a realistic environment for training RAF systems to handle granular materials, including robots, sensors, and various food items with distinct physical characteristics. We evaluate VERAGMIL by training three imitation learning models—BC, BC-RNN, and BCQ—on granular scooping and transporting tasks using both VR interface and 3D space mouse demonstrations, comparing them with a human-expert baseline. The models are assessed on success rate, spillage, generalization to unseen food items, and task completion time. Results show that VR-based demonstrations significantly outperform 3D space mouse data, with BCQ achieving the best overall performance, particularly in reducing spillage and approaching human performance. These findings underscore the effectiveness of our framework for training RAF systems in granular material handling. The code for our framework is publicly available at: https://github.com/AmanuelErgogo/VERAGMIL.git.

Due to recent advancements in computation, sensing, and hardware, robots are increasingly being adopted in human environments [1, 2]. Assistive robots, in particular, help individuals with daily tasks [3, 4]. Robot-Assisted Feeding (RAF) systems are an essential component of assistive robotics, providing critical support for individuals with

The publication was created within the project of the Minister of Science and Higher Education "Support for the activity of Centers of Excellence established in Poland under Horizon 2020" on the basis of the contract number MEiN/2023/DIR/3796. This project has received funding from the EU's Horizon 2020 research and innovation programme under grant agreement No 857533. This publication is supported by Sano project carried out within the International Research Agendas programme of the Foundation for Polish Science, co-financed by the EU Regional Development Fund.

¹ SANO Centre for Computational Personalized Medicine, Krakow, Poland. amanuel.ergogo@gmail.com

² Department of Engineering for Innovation Medicine, University of Verona, Verona, Italy.

³ Department of Computer Science and Engineering, University of South Florida, Tampa, United States. aergogo@usf.edu

<span id="page-0-0"></span>Fig. 1. VERAGMIL environment showing robotic arm (Ufactory xArm7) scooping and transporting various granular food items with different shapes, sizes, and physical properties, demonstrating adaptive manipulation of our robot-assisted feeding system.

disabilities, the elderly, and those with motor impairments to eat independently [5, 6]. Since eating is one of the most challenging tasks for these users, RAF boosts their sense of independence and significantly reduces the time caregivers spend on feeding, which is often one of the most time-consuming caregiving tasks [7, 8]. RAF systems assist in scooping, transporting, and transferring food [9, 10]. While rigid and semi-solid foods are well-handled, granular materials like rice and pasta remain challenging due to their unpredictable behavior [9, 11]. Precision and reliability are critical in these tasks, but existing RAF systems that excel with rigid foods struggle with the complexities of handling loose, particulate materials.

The difficulty in manipulating granular materials arises from their ability to behave both as solids and as fluids, depending on external forces [1, 12]. In contrast to rigid food items, which respond predictably to manipulation, granular materials exhibit nonlinear behavior, making them more challenging to handle for robotic systems [13]. These problems are particularly pronounced in RAF, where even minor inaccuracies can lead to significant spillage, reducing the reliability and usability of the system [11, 12].

Previous research in RAF has focused mainly on handling rigid or semi-solid foods, leaving the challenge of granular material manipulation relatively underexplored [5, 7]. Techniques that work well with rigid food items do not translate effectively to granular foods, where dynamic control and adaptability are necessary [9, 14]. Due to the complex and highly variable properties of granular foods, it is challenging to manually define effective control strategies with the required level of adaptability and robustness.

^{© 2025} IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. DOI: https://doi.org/10.1109/ IROS60139.2025.11247362

A promising approach to avoid manual definition of controllers is Learning from Demonstration (LfD) or Imitation Learning (IL), where a robot learns to perform tasks from human demonstrations [11, 14]. This approach enables the robot to capture how humans interact with granular materials, which can then be used to train the robot to scoop and transport food. However, collecting high-quality human demonstrations for such tasks can be labor intensive and costly in physical environments, especially when a large amount of data is needed for training [14]. Although some simulation environments have been proposed [6, 15], they are typically limited to rigid food items and lack an intuitive interface that provides 3D visual feedback and 3D control for effective task demonstration [6].

To address the limitations of current RAF systems in handling granular food items, we present VERAGMIL, a VRbased framework supporting high-performance simulation [15]. Our framework provides photo-realistic and accurate environment for RAF, supporting a variety of granular food items manipulation (as shown in Figure [1\)](#page-0-0), a virtual reality interface for capturing dexterous human demonstrations, and the training of adaptive LfD and IL manipulation policies.

In summary, VERAGMIL introduces an intuitive and efficient approach for training RAF systems to manipulate granular food items. Our key contributions include:

• We present a novel VR-based simulation framework for granular food handling in RAF. The framework features a high-fidelity physics simulation environment and a virtual reality interface designed to capture precise human demonstrations. The complete system is publicly available at: https://github.com/AmanuelErgogo/VERAGMIL.git.
• We evaluate the proposed framework by training three IL models (BC, BC-RNN, BCQ) with the acquired human demonstrations on granular scooping and transport tasks, benchmarking them against a human baseline and assessing their generalization to unseen food items.
• We evaluate the influence of the demonstration acquisition setup, comparing VR-based and 3D space mousetrained policies, showing VR-based policies achieve near-human performance and better generalization.

A. Robot-Assisted Feeding (RAF) Systems

RAF systems have been developed to assist individuals with eating tasks. Examples of such systems include commercially available solutions like Obi [16], Neater Eater [17], and Bestic [18]. In addition, there has been ongoing research in handling food items [6, 9, 12], user preferences and autonomy, which explores how users prefer to control aspects of the feeding process, such as bite timing and speed [5, 11, 19], anomaly detection and safety [7, 8, 13], and trust, acceptance, and ethics, which explores how users perceive these systems and addresses concerns related to privacy, trust, and ethics [20].

Current approaches in robot-assisted feeding systems employ a variety of manipulation strategies to handle various types of food, including solid, semi-solid, and soft materials [9, 10, 21]. Fixed trajectory planning is commonly used for predictable environments and standardized food items [22], where the robot follows pre-defined paths to acquire and transport food. While effective in structured settings, this method lacks adaptability when food shape or placement is irregular, making it unsuitable for unstructured or dynamic environments.

Adaptive trajectory planning, based on reactive control [9] and model predictive control [23, 24], adjusts a robot's movements in real-time by responding to the dynamic properties of food items. For example, visuo-haptic skewering [9] leverages real-time sensory feedback to adjust force application, allowing the robot to handle deformable food with varying textures. While effective for certain food types, these approaches rely heavily on accurately modeling the interaction between the robot and the food. This becomes especially challenging when dealing with granular food items exhibiting highly non-linear behaviors and unpredictable movement patterns, making them hard to model. The difficulty of capturing these complex dynamics makes it hard for robots to generalize their manipulation strategies, limiting their ability to handle novel or highly variable food items.

Deep Reinforcement Learning (Deep RL) can potentially enable autonomous learning of manipulation strategies in RAF. However, its application is constrained by safety concerns [25], sample inefficiency [26], and the difficulty of defining effective reward functions for complex tasks like food manipulation [27].

As detailed in the subsequent section, in the RAF context, the use of LfD could lead to effective manipulation strategies capable of handling foods with complex properties, such as granular ones.

LfD enables robots to learn complex tasks by observing human actions. In RAF, LfD helps replicate the fine motor skills required to handle diverse food textures, particularly for tasks involving granular materials where precision is crucial [28, 29]. Several studies have leveraged LfD for robotic manipulation. For instance, [30] utilizes behavioral cloning to teach robots user-specific preferences, such as bite size and feeding angle, ensuring that the robot aligns with the needs of individual users. However, this approach struggles with generalizing to new, unseen situations, as it relies on directly mimicking the demonstrations without considering the underlying dynamics of the task. In addition, contextual bandit-based learning has also been used for the real-time selection of optimal feeding strategies based on the type of food and user preferences, improving adaptability [29].

VERAGMIL supports a wide variety of LfD methods provided by the IsaacLab framework [15], but other frameworks can easily be supported. This flexibility enables the comparison of different LfD methods, as detailed in this work, and facilitates the adoption of the most effective technique for the specific RAF scenario.


<span id="page-2-0"></span>Fig. 2. Overview of the VERAGMIL framework, consisting of a VR interface and a RAF simulation environment with robot platforms, granular food items, and sensors.

The effectiveness of LfD in these tasks relies heavily on the quality of the demonstrations. Since poor-quality demonstrations lead to inefficient performance of trained policies, especially in tasks that require fine motor control [31, 32]. Traditional methods for collecting demonstrations, such as joysticks, 3D space mice, haptic interface devices and camera-based systems, have limitations. Joysticks and haptic interfaces provide intuitive control but lack 3D visual feeback and precision, while tools like the 3D space mice offer more degrees of freedom but still require considerable operator skill [32]. Camera-based systems and wearable sensors offer more natural data capture but require complex acquisition setups, not always easy to integrate into the RAF scenario and which may not capture accurately fine-grained movements necessary for granular material handling [32].

Virtual Reality (VR) has emerged as an alternative approach, offering precise control in a 3D virtual environment [33, 34]. VR allows users to interact naturally, providing 3D visual feedback and control, enabling the capture of high-quality demonstrations that reflect the complex motions required for granular food manipulation. This approach improves the accuracy of demonstrations making it highly effective for training robots in complex tasks [35, 36]. However, these existing VR solutions for human demonstrations or robot control do not provide the level of precision required for RAF tasks, are difficult to scale due to the lack of source code, the complexity of their dependencies, or the need for external workstation.

VERAGMIL overcomes these limitations, providing a ROS-based standalone VR robot interface that supports virtual and physical demonstrations, simplifies integration, and eliminates unnecessary dependencies. VERAGMIL enables realistic and natural demonstration acquisition, even in complex RAF scenarios, by fully leveraging the capabilities of VR devices. VERAGMIL also supports other demonstration acquisition interfaces, such as 3D space mice or gamepads.

Existing granular material simulators focus on accurate constitutive models rather than real-time robotic training, making them less suitable for the RAF context. While frameworks like Assistive Gym and Assistive VR Gym provide physics-based simulations for assistive tasks such as feeding, they lack GPU acceleration, which is essential for simulating granular materials like rice or cereals that require real-time adaptation [37, 38]. Without GPU acceleration, simulations must simplify item count, interactions, and rendering realism, limiting the quality of data collected for granular interactions and reducing their effectiveness for RAF.

VERAGMIL offers a GPU-accelerated high-fidelity physics simulation that can accurately simulate the operating conditions of a RAF system. The proposed approach is based on IsaacSim which ensures the modularity and extensibility of the proposed framework [15]. While IsaacSim supports the simulation of granular materials, we provide an integrated environment that supports a large set of granular foods and combines all the essential components, such as sensors, robots and end-effectors, tailored for the RAF application.

VERAGMIL addresses the complexities of RAF, particularly for granular food manipulation, by combining a highperformance simulation environment with a VR interface. This system facilitates human demonstrations and imitation learning model training for efficient granular food handling tasks. As shown in Figure [2,](#page-2-0) the system consists of three key components: the simulation environment (described in Section [III-A\)](#page-2-1), the VR control interface (see Section [III-B\)](#page-3-0), and imitation learning frameworks (detailed in Section [III-C\)](#page-3-1).

The VERAGMIL simulation environment is developed as an Omniverse extension, so as to exploit the IsaacLab framework that provides high-fidelity, GPU-accelerated simulations and robust frameworks for RAF [15]. As illustrated


TABLE I PHYSICAL PROPERTIES OF FOOD ITEMS

<span id="page-3-2"></span>

in Figure [2,](#page-2-0) VERAGMIL framework supports various robots, end-effectors, granular food items and sensors.

1) Granular Food Items: As shown in Figure [1,](#page-0-0) the dynamic behavior of different granular food items is simulated using the NVIDIA PhysX 5 physics engine, providing GPUaccelerated real-time simulation. We modeled the granular material using a set of rigid bodies implemented with the Position Based Dynamics (PBD) approach [39]. This allows granular items, such as rice and peas, to respond naturally to external forces during manipulation tasks like scooping.

As shown in Table [I,](#page-3-2) each granular material is modeled with carefully selected physical parameters, such as shape, size distribution, density, and both static and dynamic friction coefficients, as well as restitution [40]. These parameters, described with Universal Scene Description (USD) format, are crucial for accurately replicating the behavior of these materials during RAF tasks, such as scooping and transporting [41]. For instance, foods with smaller sizes and less friction, such as rice, require more precise control and force application to minimize spillage. In contrast, larger items like beans or peas can be manipulated more easily but still demand adaptive control due to their tendency to roll or shift. To ensure realistic interactions, we use Signed Distance Field (SDF) mesh colliders tailored to each food item's geometry, enabling accurate interactions with the robot's end effector [42]. By capturing the distinct characteristics of each granular item, the simulation environment allows the robot to adapt its scooping and transporting strategies based on the food's behavior, improving overall task performance in RAF.

2) Robotic Platforms: The VERAGMIL framework supports different robotic systems , including the Ufactory xArm7 and Unitree G1 humanoid, and customizable endeffectors to address various tasks in the RAF pipeline.

In our work, we consider the setup shown in Figure [4,](#page-4-0) where the xArm7 robotic manipulator is equipped with a standard spoon as end-effector for the manipulation of granular food. In this scenario, the 6-DoF control command coming from the VR interface (see Section [III-B\)](#page-3-0) is used to calculate desired end-effector at 60Hz and transformed into 7-DoF joint commands via task-space control using inverse kinematics [15]. A joint-space PD controller operating at 1000Hz then processes these commands to control the torque at each joint. This high-frequency control approach through the VR interface ensures smooth and precise manipulation of granular food items.

<span id="page-3-3"></span>Fig. 3. System architecture showing the VR interface, simulation environment and the proposed control flow.

<span id="page-3-0"></span>B. VR Interface

The VR Interface allows capturing high-quality human demonstrations of food handling tasks, such as scooping and transporting granular items, within a high-fidelity simulation environment. Natively supporting the Oculus Quest 2 system, the VR interface allows human operators to perform these tasks naturally and intuitively, providing an immersive experience. As shown in Figure [3,](#page-3-3) the movements of the human demonstrator are captured and tracked using the headset's vision system. The 6-DoF poses of both the right and left controllers, along with the headset, are transformed into the simulation's global frame, filtered to remove high-frequency jerks, and transmitted to the simulation environment at 100 Hz to compute the desired end-effector pose.

In addition to tracking, the interface streams RGB-D data from the simulation at 30Hz, providing real-time visual feedback to the operator, essential for adjusting actions while manipulating granular materials. The interface also allows the operator to control the simulation environment, including selecting relative or absolute control commands, adjusting command frequency, and scaling to ensure synchronization with the simulation's stepping speed, while inverse kinematics decimation ensures smooth control.

<span id="page-3-1"></span>C. Data Collection and Training

We collect demonstrations for scooping and transporting granular food items using two interfaces: the proposed VRbased interface for natural and precise control and a 3D space mouse for comparison [43]. The system captures multimodal data to train imitation learning models.

Each demonstration contains a recording of both proprioceptive and exteroceptive data, including the robot's 7- DoF joint positions (sjoint), end-effector pose (see), and desired post-scoop pose (sdesired). In addition, it captures RGB-D images (camerargbd) from a fixed overhead camera and relative movements of the end effector (aee) to ensure accurate task execution.

In both acquisition setups used for demonstrations, namely VR-based and 3D space mouse, we consider four different types of granular foods, i.e. rice, buckwheat, barley, and pasta. Each demonstration lasts 14-19 seconds (840-1,140 time steps at 60Hz). We collect 30 demonstrations per food



item (120 per dataset) from a single human demonstrator, totaling 28-38 minutes, with overall collection spanning 42- 76 minutes including resets.

Given a collected observation-action pair dataset D = {(sjoint, see, sdesired, camerargbd),(aee)}. The objective of IL is to find a policy π(a|s) that minimizes the loss between the demonstrated actions and the predicted actions by the model. For a comprehensive evaluation of imitation learning methods in RAF, we selected three methods: Behavioral Cloning (BC) [44], Recurrent Neural Networks (BC-RNN) [45], and Batch-Constrained Q-learning (BCQ) [46] which are designed to handle varying levels of complexity in the dataset. In BC, the robot learns the policy directly by minimizing the loss:

$$\mathcal{L}(\pi) = \mathbb{E}_{(s,a) \sim \mathcal{D}} \left[ \|\pi(s) - a\|^2 \right] \quad (1)$$

where D is the demonstration dataset, and a represents the demonstrated actions. BC, while straightforward, suffers from compounding errors in dynamic environments like granular food manipulation.

To handle this, BC-RNN extends BC by incorporating a recurrent structure, modeling the temporal dependencies between successive observations. The policy now accounts for a sequence of states st, st−1, . . . , st−n, where n is the history length, allowing it to generate actions based on past observations:

$$\pi(a_t | s_t, h_{t-1}) = f(s_t, h_{t-1}) \quad (2)$$

where ht−¹ is the hidden state from the previous timestep, allowing the model to account for time dependencies critical in continuous tasks.

BCQ introduces a reinforcement learning-based framework, where the robot learns not only to imitate but also to optimize actions based on future rewards. BCQ incorporates a Q-function Q(s, a) that estimates the future reward of an action, enabling the policy to take actions that maximize the Q-value:

$$\pi(a|s) = \arg \max_a Q(s, a) \quad (3)$$

By integrating these three methods, our framework evaluates how effectively each method learns policies for granular food handling, comparing their adaptability and performance in dynamic environments.

We assess the performance of the three considered imitation learning models, BC, BC-RNN, and BCQ, on granular material manipulation tasks in our RAF environment. In addition, we compare these models against a human operator baseline to further validate the learned policies. The tasks involve the scooping and transporting of granular materials such as rice, buckwheat, barley, and pasta, each presenting distinct physical properties such as size, shape, friction, and density, as reported in Table [I.](#page-3-2) These selected granular foods provide the basis for assessing the adaptation capabilities of the considered imitation learning methods, as well as showing the performance gap between the learned policies and human dexterity.

<span id="page-4-0"></span>Fig. 4. Experimental setup for evaluating VERAGMIL. The xArm7 manipulator robot scoops and transports granular food items, moving from the pre-scoop pose to the desired post-scoop pose, while monitoring spillage.

A. Experimental Setup

The setup involves an xArm7 robot arm with a custom spoon end effector designed to handle granular food items, as visible in Figure [4.](#page-4-0) The curvature and size of the spoon were optimized to reflect the standard tools used in real world feeding tasks. The robot's task is to scoop and transport food items from a container to a post-scoop pose without spilling. We selected rice, buckwheat, barley, pasta, beans, and flakes for their distinct flow, rolling, and scattering properties. This provided a comprehensive evaluation of the robot's ability to adapt to different material behaviors.

As described in Section [III-C,](#page-3-1) each model was trained on two datasets consisting of 30 expert demonstrations for each food item (120 per dataset). In the first scenario, human operators controlled the robot's end effector through the VR Interface (VRI), providing natural and intuitive demonstrations of the scooping and transporting tasks. In the second scenario, demonstration data were collected using a 3D space Mouse (3DM), allowing for a comparison of the impact of different input devices on training performance.

The food items used in the demonstrations included rice, buckwheat, barley, and pasta, representing a range of granular materials with varying physical properties. Meanwhile, peas, beans, and flakes were excluded from the training dataset and reserved for evaluating the generalization capability of the models on unseen materials.

All experiments were performed on a computer with 16GB RAM, an Intel Core i7-14700K CPU, and an RTX 4060 GPU. This hardware configuration does not impose particularly demanding specifications, demonstrating how VERAGMIL does not introduce entry barriers for the development of advanced RAF systems.

B. Evaluation Metrics

The performance of each model was evaluated based on four metrics: success rate, spillage, generalization, and task completion time. Success rate was defined as the percentage of tasks completed with less than 5% of the total volume spilled, ensuring acceptable precision in the manipulation

<span id="page-5-0"></span>TABLE II SUCCESS RATES FOR THE DIFFERENT MATERIALS (%). VRI REFERS TO

DEMONSTRATIONS COLLECTED IN VIRTUAL REALITY, AND 3DM REFERS TO DEMONSTRATIONS USING A 3D SPACE MOUSE.

process. Spillage was measured by the volume of food lost, approximated as the product of the number of food pieces lost during the task and the size of each lost food item. Generalization was assessed by testing the models on unseen food items—such as peas, beans, and flakes—that were not part of the training dataset, to determine how well the models could handle new materials. Finally, task completion time was measured as the duration from the robot's initial pre-scoop pose to the post-scoop pose (see Figure [4\)](#page-4-0), with the task considered complete when the robot successfully transferred the material while spilling less than 5% of the total volume. These metrics provided a comprehensive evaluation of the models' performance in terms of accuracy, adaptability, and efficiency.

V. RESULTS

The results in Table [II](#page-5-0) show that BCQ consistently outperformed both BC and BC-RNN across all scenarios. In the VRI scenario, BCQ achieved an average success rate of over 88% on food items seen during training and more than 68% on unseen items, approaching the human success rate of 95% and demonstrating strong generalization capabilities. In comparison, while BC-RNN performed well on seen items with an average success rate of 84.5%, it experienced performance drop when applied to unseen items, with an average decrease of approximately 30.95%. BC, although performing reasonably well on seen items with an average success rate of 76.25%, exhibited decline in performance on unseen items, achieving only a 40% success rate in the 3DM scenario. This highlights the superior adaptability of BCQ, particularly in handling unseen food items.

Considering the results in terms of spillage, as shown in Table [III,](#page-5-1) BCQ outperformed BC and BC-RNN, demonstrating a 20% reduction in spillage across all materials in the VRI scenario which is nearly three times the spillage of human demonstrations. This was particularly evident with larger items such as pasta, where BCQ's ability to dynamically adjust actions based on material properties resulted in significantly lower spillage rates. Last row of Table [III](#page-5-1) further highlights BCQ's stable task completion times across all materials in the VRI scenario, completing tasks slightly slower than BC but with considerably less spillage.

In contrast, the 3DM scenario, detailed in Table [III,](#page-5-1) showed an increase in spillage across all models. The chal-

<span id="page-5-1"></span>TABLE III SPILLAGE (AVERAGE NUMBER OF PIECES SPILLED) AND TASK COMPLETION TIME (SECONDS) ON FAMILIAR MATERIALS. VRI REFERS TO DEMONSTRATIONS COLLECTED IN VIRTUAL REALITY, AND 3DM REFERS TO DEMONSTRATIONS USING A 3D SPACE MOUSE.

lenge of precise manipulation using the 3D space mouse was evident, with BC experiencing the largest increase in spillage. Although task completion times were generally faster for all models in the 3DM scenario, this was accompanied by a noticeable decline in precision, particularly for BC, where spillage increased significantly.

Across both VRI and 3DM scenarios, as shown in Table [II,](#page-5-0) VR-based demonstrations consistently resulted in higher success rates and lower spillage. Specifically, BCQ demonstrated a 15% improvement in success rate and a 10% reduction in spillage when trained on VR-based demonstrations compared to data collected using the 3D space mouse. VR-based training improved all models, with BC—previously the weakest in 3DM—gaining 16% success, just 4% below the bestperforming model, BCQ-3DM 68%, and 25% below human 3DM performance 89%. This trend, evident across all models, underscores the effectiveness of VR as a demonstration tool, as it captures more natural and intuitive human motions, leading to improved learning outcomes for the robot.

Overall, BCQ excelled across all metrics, particularly in VRI, demonstrating strong generalization. VR-based demonstrations consistently improved task success and precision, reinforcing the importance of intuitive interface. Further details on the properties of the materials and experimental setup are provided in the code repository.

VI. DISCUSSIONS

The results demonstrate the impact of different imitation learning models and demonstration methods on the performance of granular food item handling tasks. Across both VRI and 3DM scenarios, BCQ consistently outperformed BC and BC-RNN in terms of success rate, spillage, and generalization. This is in line with previously published results that consider different robotic applications and require simpler interaction with the environment [43, 45]. We plan to extend our validation to additional imitation learning methods and also adopt demonstration-free reinforcement learning techniques to understand which solution is more effective for RAF.

The VR-based demonstrations proved to be a more effective data collection method, resulting in higher success rates and reduced spillage across all models. The VR-trained models exhibited improved adaptability to unseen granular materials, such as peas, beans, and flakes, even for simpler models like BC. The results obtained confirm those of previous works [35, 38]; but we plan to conduct an extended user study, including objective usability and workload evaluation, to more clearly confirm the effectiveness of the proposed VR

• interface. The use of BC-RNN provided a notable advantage in capturing sequential dependencies in granular food manipulation. However, its overall performance was inferior to that of BCQ, particularly in generalization tasks. As expected, BC performed well on familiar food items but struggled to generalize to unseen materials, highlighting the importance of dynamic learning strategies in imitation learning. Although previous studies have focused mainly on solid foods [30] or small operating spaces [28], our results extend these findings to a more complex environment. Furthermore, task completion times were slightly faster in the 3DM scenario compared to VRI. However, this speed came at the cost of precision, with increased spillage in the 3DM scenario, especially in BC models. This trade-off between speed and precision suggests that while both input methods are viable, VR-based demonstrations offer a more balanced approach to ensure accuracy in challenging tasks related to granular food handling. To make our validation even more thorough and accurate, we are actively working to extend the experimental validation to consider the real environment, adopting techniques to reduce the gap between simulation and reality [47, 48]. We also plan to improve the simulation realism, including more granular foods and their interaction with liquids of different densities and viscosities, in order to simulate soups or foods with mixed compositions.
• VII. CONCLUSION This paper presents VERAGMIL, a VR-based imitation learning framework for granular food manipulation in RAF, which can be adapted to other granular manipulation skills with small task-specific configurations, featuring a highfidelity simulation environment and an intuitive VR demonstration interface. We evaluated VERAGMIL by training imitation learning models on scooping and transporting tasks, using VR-based and 3D space mouse demonstrations, and comparing them to a human-expert baseline. VERAGMIL improves success rates and reduces spillage, with models approaching human performance. Although spillage is significantly reduced, fine-grained control over small portions of granular materials requires improvement. Future work will focus on sim-to-real transfer, real-world testing, and user studies. REFERENCES [1] A. Nanavati, P. Alves-Oliveira et al., "Design principles for robotassisted feeding in social contexts," ACM/IEEE International Conference on Human-Robot Interaction, 2023. [2] G. Lancioni, L. Desideri, N. N. Singh, M. O'Reilly, and J. Sigafoos, "Technology options to help people with dementia or acquired cognitive impairment perform multistep daily tasks: A scoping review," Journal of Enabling Technologies, 2021. [3] S. P. A. and A. Mohan, "Design and simulation of an autonomous indoor robot for elderly assistance," IEEE 2nd Mysore Sub Section International Conference (MysuruCon), 2022. [4] Y. Yamazaki, M. Ishii, T. Ito, and T. Hashimoto, "Frailty care robot for elderly and its application for physical and psychological support," Journal of Advanced Computational Intelligence and Intelligent Informatics, 2021. [5] T. Bhattacharjee, E. Gordon et al., "Is more autonomy always better? exploring preferences of users with mobility impairments in robotassisted feeding," ACM/IEEE International Conference on Human-Robot Interaction, 2020. [6] Z. M. Erickson, V. Gangaram et al., "Assistive gym: A physics simulation framework for assistive robotics," IEEE International Conference on Robotics and Automation (ICRA), 2019. [7] D. Park, Y. Hoshi et al., "Toward active robot-assisted feeding with a general-purpose mobile manipulator: Design, evaluation, and lessons learned," arXiv, 2019. [8] E. Gordon et al., "Leveraging post hoc context for faster learning in bandit settings with applications in robot-assisted feeding," IEEE International Conference on Robotics and Automation, 2020. [9] P. Sundaresan, S. Belkhale, and D. Sadigh, "Learning visuo-haptic skewering strategies for robot-assisted feeding," Conference on Robot Learning, 2022. [10] J. Grannen, Y. Wu, S. Belkhale, and D. Sadigh, "Learning bimanual scooping policies for food acquisition," in Conference on Robot Learning. PMLR, 2023, pp. 1510–1519. [11] E. Gordon, X. Meng, T. Bhattacharjee et al., "Adaptive robot-assisted feeding: An online learning framework for acquiring previously unseen food items," IEEE/RJS International Conference on Intelligent Robots and Systems (IROS), 2019. [12] P. Parikh, R. Trivedi, J. Dave et al., "Design and development of a low-cost vision-based 6 dof assistive feeding robot for the aged and specially-abled people," Journal of the Institution of Electronics and Telecommunication Engineers, 2023. [13] D. Park, Y. Hoshi et al., "A multimodal anomaly detector for robotassisted feeding using an lstm-based variational autoencoder," IEEE Robotics and Automation Letters, 2017. [14] Z. M. Erickson, Y. Gu et al., "Assistive vr gym: Interactions with real people to improve virtual assistive robots," IEEE International Symposium on Robot and Human Interactive Communication, 2020. [15] NVIDIA, "Isaacsim and isaaclab," 2020. [Online]. Available: https://developer.nvidia.com/isaac-sim [16] DESiN LLC, "Obi - The Dining Companion," http://www.desinllc. com/products/obi, 2023, [Online; accessed 12-September-2024]. [17] Neater Solutions, "Neater Eater - Robotic Eating Aid," http://www. neatersolutions.com/neater-eater.html, 2023, [Online; accessed 12- September-2024]. [18] Camanio Care, "Bestic - Eating Made Easy," http://www.camanio. com/products/bestic, 2023, [Online; accessed 12-September-2024]. [19] J. Ondras, A. Anwar et al., "Human-robot commensality: Bite timing prediction for robot-assisted feeding in groups," Conference on Robot Learning, 2022. [20] M. N. C. Nickelsen, "Imagining and tinkering with assistive robotics in care for the disabled," Paladyn J. Behav. Robotics, 2019. [21] K. Takahashi, W. Ko, A. Ummadisingu, and S.-i. Maeda, "Uncertaintyaware self-supervised target-mass grasping of granular foods," in 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021, pp. 2620–2626. [22] G.-N. Zhu, Y. Zeng, Y. S. Teoh, E. Toh, C. Y. Wong, and I.-M. Chen, "A bin-picking benchmark for systematic evaluation of roboticassisted food handling for line production," IEEE/ASME Transactions on Mechatronics, vol. 28, no. 3, pp. 1778–1788, 2022. [23] P. Wang, H. Deng, J. Zhang, L. Wang, M. Zhang, and Y. Li, "Model predictive control for connected vehicle platoon under switching communication topology," IEEE Transactions on Intelligent Transportation Systems, vol. 23, pp. 7817–7830, 2022. [24] R. Grandia, F. Jenelten, S. hua Yang, F. Farshidian, and M. Hutter, "Perceptive locomotion through nonlinear model-predictive control," IEEE Transactions on Robotics, vol. 39, pp. 3402–3421, 2022. [25] M. K. Tejas Kumar Shastha and A. Graser, "Application of reinforcement learning to a robotic drinking assistant," Robotics, vol. 9, no. 1, pp. 1–15, 2019. [26] S. Alighanbari and N. Azad, "Deep reinforcement learning with nmpc assistance nash switching for urban autonomous driving," IEEE Transactions on Intelligent Vehicles, vol. 8, pp. 2604–2615, 2023. [27] T. Xie, S. Zhao, C. H. Wu, Y. Liu, Q. Luo, V. Zhong, Y. Yang, and T. Yu, "Text2reward: Reward shaping with language models for reinforcement learning," in The Twelfth International Conference on Learning Representations, 2023. [28] P. Sundaresan, S. Belkhale, and D. Sadigh, "Learning visuo-haptic skewering strategies for robot-assisted feeding," in 6th Annual Conference on Robot Learning, 2022.

• [29] E. Gordon, X. Meng, T. Bhattacharjee, M. Barnes, and S. Srinivasa, "Adaptive robot-assisted feeding: An online learning framework for acquiring previously unseen food items," in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019, pp. 9659– 9666. [30] T. Bhattacharjee, E. K. Gordon, R. Scalise, M. E. Cabrera, A. Caspi,
• M. Cakmak, and S. S. Srinivasa, "Is more autonomy always better? exploring preferences of users with mobility impairments in robotassisted feeding," in 2020 15th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 2020, pp. 181–190. [31] V. Tangkaratt, B. Han, M. E. Khan, and M. Sugiyama, "Vild: Variational imitation learning with diverse-quality demonstrations," International Conference on Machine Learning, pp. 9407–9417, 2019. [32] B. Hao, R. Jain, T. Lattimore, B. V. Roy, and Z. Wen, "Leveraging demonstrations to improve online learning: Quality matters," International Conference on Machine Learning, pp. 12 527–12 545, 2023. [33] Z. Makhataeva and H. A. Varol, "Augmented reality for robotics: A review," Robotics, vol. 9, no. 2, p. 21, 2020. [34] E. Natonek, L. Flueckiger, T. Zimmerman, and C. Baur, "Virtual reality: an intuitive approach to robotics," in Telemanipulator and Telepresence Technologies, vol. 2351. SPIE, 1995, pp. 260–270. [35] A. Ergogo, D. Dall'Alba, and P. Korzeniowski, "Hilotel: Virtual reality robot interface-based human-in-the-loop task execution and learning in the physical world through its digital twin," in 2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), 2024, pp. 681–682. [36] S. B. i. Badia, P. A. Silva, D. Branco, A. Pinto, C. Carvalho,
• P. Menezes, J. Almeida, and A. Pilacinski, "Virtual reality for safe testing and development in collaborative robotics: challenges and perspectives," Electronics, vol. 11, no. 11, p. 1726, 2022. [37] Z. Erickson, V. Gangaram, A. Kapusta, C. K. Liu, and C. C. Kemp, "Assistive gym: A physics simulation framework for assistive robotics," in 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020, pp. 10 169–10 176. [38] Z. Erickson, Y. Gu, and C. C. Kemp, "Assistive vr gym: Interactions with real people to improve virtual assistive robots," in 2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN). IEEE, 2020, pp. 299–306. [39] M. Muller, M. Macklin, N. Chentanez, S. Jeschke, and T. Kim, "De- ¨ tailed rigid body simulation with extended position based dynamics," Computer Graphics Forum, vol. 39, pp. 101–112, 12 2020. [40] M. Macklin, M. Muller, and N. Chentanez, "Xpbd: position-based ¨ simulation of compliant constrained dynamics," in Proceedings of the 9th International Conference on Motion in Games, 2016, pp. 49–54. [41] D. Dall'Alba, M. Nasket, S. Kaminska, and P. Korzeniowski, "Ff-srl: ´ High performance gpu-based surgical simulation for robot learning," in 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024, pp. 8378–8384. [42] Y. Li, Y. Zhang, A. Razmjoo, and S. Calinon, "Representing robot geometry as distance fields: Applications to whole-body manipulation," in Proc. IEEE Intl Conf. on Robotics and Automation (ICRA), 2024, pp. 15 351–15 357. [43] Y. Zhu, J. Wong, A. Mandlekar, R. Mart´ın-Mart´ın, A. Joshi,
• K. Lin, S. Nasiriany, and Y. Zhu, "robosuite: A modular simulation framework and benchmark for robot learning," in arXiv preprint arXiv:2009.12293, 2020. [44] D. A. Pomerleau, "Alvinn: An autonomous land vehicle in a neural network," Advances in neural information processing systems, vol. 1, 1988. [45] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni,
• L. Fei-Fei, S. Savarese, Y. Zhu, and R. Mart´ın-Mart´ın, "What matters in learning from offline human demonstrations for robot manipulation," in arXiv preprint arXiv:2108.03298, 2021. [46] S. Fujimoto, D. Meger, and D. Precup, "Off-policy deep reinforcement learning without exploration," in International conference on machine learning. PMLR, 2019, pp. 2052–2062. [47] P. M. Scheikl, E. Tagliabue, B. Gyenes, M. Wagner, D. Dall'Alba,
• P. Fiorini, and F. Mathis-Ullrich, "Sim-to-real transfer for visual reinforcement learning of deformable object manipulation for robotassisted surgery," IEEE Robotics and Automation Letters, vol. 8, no. 2, pp. 560–567, 2022. [48] J. Kaleta, D. Dall'Alba, S. Płotka, and P. Korzeniowski, "Minimal data requirement for realistic endoscopic image generation with stable diffusion," International journal of computer assisted radiology and surgery, vol. 19, no. 3, pp. 531–539, 2024.