2021-12-13 · 105 citations · club pick

Learning Generalizable Vision-Tactile Robotic Grasping Strategy for Deformable Objects via Transformer

Yunhai Han, Kelin Yu, Rahul Batra, Nathan Boyd, Chaitanya Mehta, Tuo Zhao, Yu She, Seth Hutchinson, Ye Zhao

Dexterous manipulation Tactile & force sensing

No peer-reviewed venue on record yet. 105 citations, 4 of them influential, as of the last refresh.

Abstract

Reliable robotic grasping, especially with deformable objects such as fruits, remains a challenging task due to underactuated contact interactions with a gripper, unknown object dynamics and geometries. In this study, we propose a Transformer-based robotic grasping framework for rigid grippers that leverage tactile and visual information for safe object grasping. Specifically, the Transformer models learn physical feature embeddings with sensor feedback through performing two pre-defined explorative actions (pinching and sliding) and predict a grasping outcome through a multilayer perceptron (MLP) with a given grasping strength. Using these predictions, the gripper predicts a safe grasping strength via inference. Compared with convolutional recurrent networks, the Transformer models can capture the long-term dependencies across the image sequences and process spatial-temporal features simultaneously. We first benchmark the Transformer models on a public dataset for slip detection. Following that, we show that the Transformer models outperform a CNN+LSTM model in terms of grasping accuracy and computational efficiency. We also collect a new fruit grasping dataset and conduct online grasping experiments using the proposed framework for both seen and unseen fruits. {In addition, we extend our model to objects with different shapes and demonstrate the effectiveness of our pre-trained model trained on our large-scale fruit dataset. Our codes and dataset are public on GitHub.

Ten-minute slide kit

Six slides is the whole talk: what was broken, what people tried, what these authors did, what the numbers say, where it falls over, and the sentence people should remember.

SLIDE 1

The problem

R OBOT manipulation has been widely used in industries for decades, but mostly for repetitive tasks in structured environment where there is little uncertainty or contact deformation in manipulated objects. For the tasks where object contact parameters are prone to vary, such as fruit grasping, they are still challenging for robotic systems [\[1\]](#page-10-0). Loose grips with small grasping forces can cause objects to slip, while large grasping forces can cause

SLIDE 2

What came before

Not recoverable from the parsed text. Read this section in the paper yourself.

SLIDE 3

The method

In this section, we describe the details of the grasping framework and each Transformer model. In order to give robots the ability to estimate the safe grasping force, we first let the robot obtain physical information about the target objects (fruits in this work) by performing two explorative actions, *pinching* and *sliding*, on the objects. To avoid any potential damage, these actions have minimum interaction with the objects. To monitor the interactions and record the data, the robot is equipped with two…

SLIDE 4

What they measured

In this section, we present our experiments using the Transformer models. In this section, we evaluate our framework with unseen irregular objects (i.e., corn and banana) during online experiments, the same as Sec. Unlike kiwi, corn and bananas are new objects with irregular shapes and different contact surface textures, as shown in Fig. 14. Note that, we use a new gripper in this section for this experiment, which has almost exactly the same mechanical properties as the previous one. *1) Experiment Evaluation:*…

SLIDE 5

Where it breaks

Our experiments demonstrate that the Transformer models can enable robotic grasping tasks in both the object classi- <span id="page-9-2"></span><sup>5</sup>https://drive.google.com/file/ fication and robot control domain. The results indicate that they outperform traditional models, such as CNN+LSTM, for classification tasks like slip detection and grasping outcome prediction. In addition, our Transformer-based grasping framework is able to select the grasping strength to safely grasp fruits with varying hardness…

SLIDE 6

One-line takeaway

This study proposes a transformer-based robotic grasping framework for rigid grippers that leverage tactile and visual information for safe object grasping and shows that the transformer models outperform a CNN + LSTM model in terms of grasping accuracy and computational efficiency.

Assembled from the paper's own PDF, parsed with its layout intact, 69,132 characters of it, then split on the paper's own section headings. Extractive, not generated: every sentence here is lifted from the paper. Check it before you present it.

Presented at

Saturday, August 22, 2026

Saturday Robotics, session 25

Saturday Robotics x Trossen @ Mission Robotics & World Models Reading Club 25: Contact-Rich Robot Learning Human Videos & Tactile. SF 8/22

San Francisco, CA