Simultaneous Estimation of Manipulation Skill and Hand Grasp Force from Forearm Ultrasound Images
Keshav Bimbraw, Srikar Nekkanti, Daniel B. Tiller II, Mihir Deshmukh, Berk Calli, Robert D. Howe, Haichong K. Zhang · 2025
Read Simultaneous Estimation of Manipulation Skill and Hand Grasp Force from Forearm Ultrasound Images by Keshav Bimbraw, Srikar Nekkanti, Daniel B. Tiller II, Mihir Deshmukh, Berk Calli, Robert D. Howe, Haichong K. Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-00275.txt.
Introduction
Keshav Bimbraw and a team of co-authors published a 2025 study that reads hand skill and grip force from ultrasound images of the forearm. Knowing what a human hand is doing, and how hard it grips, would help robots learn from people and be controlled at a distance. Signals from the body, such as ultrasound, are a promising way to capture that information.
Data from seven participants trained deep learning models for two jobs at once. The models named the manipulation skill with about 95 percent accuracy on average. They also estimated grip force with an average error near half a newton. The results support ultrasound as a practical channel for passing human dexterity to machines.
Catalog record
| CREATOR | Keshav Bimbraw, Srikar Nekkanti, Daniel B. Tiller II, Mihir Deshmukh, Berk Calli, Robert D. Howe, Haichong K. Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-00275
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Simultaneous Estimation of Manipulation Skill and Hand Grasp Force from Forearm Ultrasound Images Keshav Bimbraw, Srikar Nekkanti, Daniel B. Tiller II, Mihir Deshmukh, Berk Calli, Robert D. Howe, Haichong K. Zhang Abstract Accurate estimation of human hand configuration and the forces they exert is critical for effective teleoperation and skill transfer in robotic manipulation. A deeper understanding of human interactions with objects can further enhance teleoperation performance. To address this need, researchers have explored methods to capture and translate human manipulation skills and applied forces to robotic systems. Among these, biosignal-based approaches, particularly those using forearm ultrasound data, have shown significant potential for estimating hand movements and finger forces. In this study, we present a method for simultaneously estimating manipulation skills and applied hand force using forearm ultrasound data. Data collected from seven participants were used to train deep learning models for classifying manipulation skills and estimating grasp force. Our models achieved an average classification accuracy of 94.87% ± \pm 10.16% for manipulation skills and an average root mean square error (RMSE) of 0.51 ± \pm 0.19 N for force estimation, as evaluated using five-fold cross-validation. These results highlight the effectiveness of forearm ultrasound in advancing human-machine interfacing and robotic teleoperation for complex manipulation tasks. This work enables new and effective possibilities for human-robot skill transfer and tele-manipulation, bridging the gap between human dexterity and robotic control. 1 Introduction With the advances in robotics and artificial intelligence (AI), robots are increasingly being used to manipulate objects not only on the factory floor but also in everyday human environments, such as homes and healthcare settings [ 1 ] . Robotic manipulation finds applications in package handling [ 2 ] , manufacturing [ 3 ] , healthcare [ 4 ] , agriculture [ 5 ] , and space [ 6 ] among others. In unstructured environments, robots must handle a diverse range of objects, requiring adaptable manipulation strategies to accomplish their tasks effectively. Effectively interacting with different objects requires varied manipulation skills tailored to their unique properties and constraints [ 7 ] . This necessity arises due to the diverse geometries, textures, and mechanical properties of objects, which influence the selection of appropriate manipulation strategies [ 8 ] . For instance, flat objects may require sliding to an edge of a surface for getting access to grasping surfaces (for example, a credit card). If a cylindrical object is positioned against a vertical surface, a picking strategy could involve pushing it towards the surface to stabilize the grasp and facilitate access to suitable grasping areas, such as edges or protrusions. Alongside selecting the correct skill, applying the appropriate amount of force during interaction is equally crucial [ 9 ] . Insufficient force can lead to a loss of control, while excessive force might damage delicate objects or destabilize the task. Thus, a robust manipulation framework must integrate both skill identification and force estimation to achieve desired performance. These capabilities are essential for enabling robots to handle diverse objects and perform complex tasks with human-like adaptability. This is particularly relevant in applications such as autonomous manufacturing [ 10 ] , assistive technologies [ 11 ] , and robot-assisted surgery [ 12 ] where task success depends on both the execution of appropriate skills and the transfer of the correct forces during object interaction. We propose an approach for simultaneously estimating manipulation skills and hand forces using ultrasound data from the forearm. By leveraging brightness mode (B-mode) ultrasound, our method addresses a central requirement shared by two primary paradigms in human-like robotic manipulation: teleoperation and Learning from Demonstrations (LfD). Both of these approaches rely on accurate skill identification and precise force estimation. Teleoperation mirrors the human operator’s movements in real time and is often employed in scenarios where precise, immediate control is necessary, such as remote surgical systems [ 13 ] or hazardous environment operations [ 14 ] . Although teleoperation is limited by communication latency and scalability issues, it remains highly useful for tasks that require continuous human oversight [ 15 ] . Wearable gloves and vision-based techniques have been used to teleoperate robotic systems [ 16 , 17 ] . However, gloves restrict natural hand movements, while vision-based systems suffer from occlusion, poor lighting conditions, and shadow artifacts. Bodily signals have been used as an alternative, with surface electromyography (sEMG) [ 18 , 19 ] , force myography (FMG) [ 20 , 21 ] , near-infrared spectroscopy (NIRS) [ 22 ] , electric impedance tomography (EIT) [ 23 ] , and ultrasound [ 24 , 25 ] all explored for robotic control. Surface electromyography suffers from inherent limitations such as high sensitivity to electrode location and issues with electrode–skin contact [ 18 ] . FMG faces susceptibility to sensor drift and signal variability due to preload inconsistencies [ 26 ] , while NIRS is sensitive to tissue thickness and ambient light interference [ 27 ] . EIT encounters challenges in image reconstruction that impact measurement sensitivity, range, and response time [ 28 ] . Ultrasound, although limited by sensor size, has shown promise for estimating hand gestures [ 29 ] , finger angles [ 30 ] , and finger forces [ 31 ] . Furthermore, ongoing efforts are focused on miniaturizing ultrasound sensors [ 32 ] and making them more wearable [ 33 ] for human–machine interfacing applications. LfD enables robots to learn task constraints and execution strategies by observing expert demonstrations [ 34 ] . Using supervised learning, LfD aligns closely with tasks where ideal behavior is difficult to formalize but can be effectively demonstrated [ 35 ] . This approach has been extensively explored for trajectory learning, policy optimization, and skill generalization in applications ranging from manufacturing to healthcare [ 36 , 37 ] . Multiple sensor data acquisition is commonly utilized to encode task-specific variables effectively. For instance, [ 38 ] integrated multimodal data (motion trajectories, stiffness profiles obtained via sEMG, and force/torque measurements) to capture correlations between position, velocity, stiffness, and force, allowing precise reproduction of tasks with combined position and force constraints. Similarly, Vasan et al. demonstrated the application of LfD to train sEMG-based prostheses using contralateral limb movements [ 39 ] . By integrating sEMG signals and joint trajectories from the intact limb, they mapped user demonstrations into a policy for controlling multi degree-of-freedom prosthetic devices. The choice between teleoperation and LfD depends on task-specific requirements. Teleoperation is advantageous for high-stakes, dynamic tasks requiring direct human intervention, whereas LfD offers scalability and autonomy by enabling robots to adapt and generalize behaviors based on prior demonstrations. Both teleoperation and LfD demand robust skill identification and force sensing, motivating our approach. Our proposed ultrasound-based approach supports both paradigms by providing real-time skill and force estimation (for teleoperation) and serving as a rich source of task demonstrations (for LfD). By focusing on motion primitives underlying manipulation tasks, rather than strict trajectory replication, we facilitate efficient transfer of human-like dexterity to robots. This capability enhances the adaptability and precision of robotic manipulation systems in dynamic and unstructured environments. 1.1 Related Works Zongxing et al. provided a comprehensive review of modalities for simultaneous gesture recognition and force assessment, including vision-based systems, surface electromyography (sEMG), ultrasound imaging, and hybrid approaches [ 40 ]