Object Recognition and Force Estimation with the GelSight Baby Fin Ray

Sandra Q. Liu, Yuxiang Ma, Edward H. Adelson · 2025

Read Object Recognition and Force Estimation with the GelSight Baby Fin Ray by Sandra Q. Liu, Yuxiang Ma, Edward H. Adelson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-14510.txt.

Introduction

Sandra Q. Liu, Yuxiang Ma and Edward H. Adelson extend a soft tactile robot finger in 2025, the GelSight Baby Fin Ray. The device mixes a camera with a soft fin shaped structure that bends on contact. Camera based touch captures force, shape and texture richly.

Earlier work showed the finger digging through clutter and sorting nuts in shells. New learning tasks test finer skill: telling nut textures apart through the shell, and estimating contact force and position. Ablation studies with popular neural networks show which design choices drive performance. Soft tactile fingers that can feel texture and force could help robots handle fragile or hidden objects where vision alone fails.

Catalog record

CREATORSandra Q. Liu, Yuxiang Ma, Edward H. Adelson
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE8 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-14510

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Object Recognition and Force Estimation with the GelSight Baby Fin Ray Sandra Q. Liu, Yuxiang Ma, Edward H. Adelson Abstract Recent advances in soft robotic hands and tactile sensing have enabled both to perform an increasing number of complex tasks with the aid of machine learning. In particular, we presented the GelSight Baby Fin Ray in our previous work [ 1 ] , which integrates a camera with a soft, compliant Fin Ray structure. Camera-based tactile sensing gives the GelSight Baby Fin Ray the ability to capture rich contact information like forces, object geometries, and textures. Moreover, our previous work showed that the GelSight Baby Fin Ray can dig through clutter, and classify in-shell nuts. To further examine the potential of the GelSight Baby Fin Ray, we leverage learning to distinguish nut-in-shell textures and to perform force and position estimation. We implement ablation studies with popular neural network structures, including ResNet50, GoogLeNet, and 3- and 5-layer convolutional neural network (CNN) structures. We conclude that machine learning is a promising technique to extract useful information from high-resolution tactile images and empower soft robotics to better understand and interact with the environments. Keywords: Tactile sensors, Fin Ray, contact sensing, object perception 1 Introduction The Fin Ray finger is inspired by fish fin deformation, where the fin bends against and complies to the force applied on its structure [ 2 ] . As a result, the Fin Ray can be implemented for different robotic tasks and, due to its simple structure, is easy to print and prototype. Although much research has focused on mechanical behavior optimization [ 3 , 4 ] , there has been sparse work on integrating sensors into Fin Rays. Fin Rays could benefit from having tactile feedback, and several works have focused on adding force sensors or using embedded cameras and machine learning to do force estimation [ 5 , 6 ] . More recently, work has been done to integrate classically rigid camera-based sensors into various soft finger designs [ 7 , 8 , 9 , 10 ] . Specifically, camera-based sensors can provide higher-resolution details, discern object geometries, and measure contact forces [ 11 ] . The original GelSight Fin Ray was able to have the passive adaptability of its compliant structure and the high-resolution tactile sensing enabled by camera sensors [ 12 ] . These capabilities allowed it to grasp various objects, track surface forces, and perform 3D reconstruction of the tactile deformation. Further mechanical improvements resulted in the GelSight Baby Fin Ray, which is more compliant, robust, and compact than its predecessor [ 1 ] . Figure 1: On the left, the GelSight Baby Fin Ray grabs a walnut; on the right is the corresponding raw tactile image [ 1 ] . We present further work on leveraging the camera-based tactile sensor in our Fin Ray structure. In this work, we implement multiple machine learning algorithms to classify textures of nuts-in-shells and do force estimation and tactile position determination along the length of the tactile sensor. 2 Related Work Machine learning techniques have been used for multiple rigid finger sensors incorporated with camera-based tactile sensors. Guo et al. and Huang et al. are able to supplement these sensors with machine learning methods to estimate properties of solid particles and liquids inside containers [ 13 , 14 ] . Other papers have used various learning methods to extract information about surface forces and 3D geometrical reconstruction of tactile deformation [ 15 , 16 , 17 ] . Still, others have done texture classification [ 7 ] . On the other hand, only a few papers have performed proprioception on soft robots, i.e. state estimation or force prediction, and have not really focused on identifying textures [ 18 , 19 , 9 ] . This work implements both object classification using texture and force estimation with a single camera sensor incorporated into a soft, compliant robot finger. 3 Hardware We utilize the hardware design from our previous work on the GelSight Baby Fin Ray [ 1 ] . A representation of this design is shown in Fig. 2 Figure 2: a) A 2D representation of the GelSight Baby Fin Ray, which shows that the camera and mirror are able to view the entire tactile sensing region. b) Exploded view of the assembly without the camera, LEDs, and diffuser [ 1 ] . 4 Experiment & Results 4.1 Nut Classification One of the original tasks performed by the GelSight Baby Fin Ray paper was nuts-in-shell classification. We perform ablation studies against the ResNet50 architecture [ 20 ] that was used in [ 1 ] and compare the results with GoogLeNet [ 21 ] , a 3-layer and a 5-layer CNN [ 22 ] , a support vector machine (SVM) using both radial basis functions (RBF) and polynomial functions [ 23 ] , and a k-nearest neighbors (KNN) classification algorithm [ 24 ] . For the Resnet, CNN, and GoogLeNet architectures, we use an unwarped, augmented image as the input, effectively getting rid of the “negative” space in the image. For both the KNN and SVM algorithms, we create a feature vector using the raw pixels in the image as the input. We collect data using images streamed from the raspberry pi camera via mjpg_streamer. Nuts are pressed along the entire tactile surface at various angles. A total of 500 images per nut type are captured. The data is split into 80% training and 20% validation. Figure 3: To classify our four classes, Almond, Brazil Nut, Pecan, and Walnut, we take the raw image, unwarp it and train it using multiple neural network architectures including ResNet50, GoogLeNet, 3- and 5-layer CNN, SVM, and KNN. Table 1: Accuracy of Nut Classification Network Overall Almonds Brazil Nuts Pecans Walnuts 3-layer CNN 73.0% 73.6 % 84.0 % 70.8 % 64.0 % 5-layer CNN 87.5% 82.1 % 87.2 % 91.7 % 89.4 % ResNet 95.6% 98.1% 96.8 % 96.9 % 91.3 % GoogLeNet 97.5% 96.2 % 98.9% 96.9 % 98.1% SVM (poly) 86.3% 83.0 % 86.0 % 79.0 % 97.0 % SVM (rbf) 79.0% 81.0 % 77.0 % 68.0 % 90.0 % KNN 73.3% 71.0 % 78.0 % 72.0 % 72.0 % Our results are shown in Table 1 . In general, the deeper networks performed better on the dataset. We believe that these results could be because deeper architectures are able to recognize more minute and complex details that might have been lost by a more shallow net or by using SVM and KNN methods. Nonetheless, all of the algorithms and neural nets performed well, indicating that the high-resolution tactile details of the nut shells are easily discernible. 4.2 Force Estimation We also establish learning-based force estimation with the Gelsight Baby Fin Ray. A cylinder or cuboid indenter and a Gelsight Baby Fin Ray are installed on a Franka Panda Hand, as shown in Fig. 4 . We program the robot hand to push indenters into the Gelsight Baby Fin Ray, while using a Nano 17 Force/Torque (ATI) sensor to record the indentation force. Concurrently, tactile images are collected via mjpg_streamer. Contacts are made at 10 mm to 50 mm along the Fin Ray length, and the range of normal force is

Read the complete text