Touch Speaks, Sound Feels: A Multimodal Approach to Affective and Social Touch from Robots to Humans.

Qiaoqiao Ren, and Tony Belpaeme · 2025

Read Touch Speaks, Sound Feels: A Multimodal Approach to Affective and Social Touch from Robots to Humans. by Qiaoqiao Ren, and Tony Belpaeme (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07839.txt.

Introduction

Ren and Belpaeme test whether robots can speak emotion through touch and sound together, the way people do. In daily life a pat or stroke arrives with contact noise, and the pairing carries feeling. Their 2025 system uses a grid of twenty-five vibrating motors synced with audio clips to deliver combined signals.

Thirty-two participants in China rated ten emotions and six social gestures given as vibration only, sound only, or both, scoring each for intensity and pleasantness. Combined signals were decoded far better than either channel alone. Each single channel still carried some emotions well, with strengths split by emotion. Gestures alone blurred into one another. The authors back multi-sense design for social robots.

Catalog record

CREATORQiaoqiao Ren, and Tony Belpaeme
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-07839

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Touch Speaks, Sound Feels: A Multimodal Approach to Affective and Social Touch from Robots to Humans. Qiaoqiao Ren, and Tony Belpaeme Abstract Affective tactile interaction constitutes a fundamental component of human communication. In natural human–human encounters, touch is seldom experienced in isolation; rather, it is inherently multisensory. Individuals not only perceive the physical sensation of touch but also register the accompanying auditory cues generated through contact. The integration of haptic and auditory information forms a rich and nuanced channel for emotional expression. While extensive research has examined how robots convey emotions through facial expressions and speech, their capacity to communicate social gestures and emotions via touch remains largely underexplored. To address this gap, we developed a multimodal interaction system incorporating a 5×5 grid of 25 vibration motors synchronized with audio playback, enabling robots to deliver combined haptic-audio stimuli. In an experiment involving 32 Chinese participants, ten emotions and six social gestures were presented through vibration, sound, or their combination. Participants rated each stimulus on arousal and valence scales. The results revealed that (1) the combined haptic-audio modality significantly enhanced decoding accuracy compared to single modalities; (2) each individual channel—vibration or sound—effectively supported certain emotions recognition, with distinct advantages depending on the emotional expression; and (3) gestures alone were generally insufficient for conveying clearly distinguishable emotions. These findings underscore the importance of multisensory integration in affective human–robot interaction and highlight the complementary roles of haptic and auditory cues in enhancing emotional communication. I Introduction Touch is a fundamental aspect of human interaction, serving as a primary channel for communication, emotional expression, and social bonding [ 1 ] . Unlike other sensory modalities such as vision or hearing, touch provides direct and immediate feedback, reinforcing trust, intimacy, and empathy in interpersonal relationships [ 2 ] . Studies in psychology and neuroscience have shown that affective touch, such as a gentle pat or a reassuring squeeze, can reduce stress, enhance well-being, and strengthen social bonds [ 3 , 1 ] . Early research on human-human affective tactile communication suggests that touch primarily conveys emotional valence and intensity, with context playing a critical role in interpretation [ 4 ] . Hertenstein et al. demonstrated that emotions such as anger, disgust, fear, gratitude, happiness, love, sadness, and sympathy could be recognized solely through touch, with recognition rates between 48% and 83%, whereas self-focused emotions (e.g., embarrassment, envy, pride) were less accurately conveyed [ 5 ] . Studies have shown that haptic behaviour alone is insufficient for distinct emotion recognition, with research identifying 23 different types of tactile behaviours (e.g., hugging, squeezing, shaking) that do not exclusively correspond to a single emotion. For instance, stroking was observed in expressions of sadness, love, and sympathy, suggesting that tactile behaviours must be interpreted alongside other features such as duration and intensity to enhance emotion decoding. [ 6 ] . As robots become increasingly integrated into daily life—from caregiving and therapy to collaborative workspaces, establishing effective communication between humans and robots is essential [ 7 ] . While current research in human-robot affective interaction has predominantly emphasized modalities such as speech [ 8 ] , facial expressions [ 9 ] , and body movement [ 10 ] , the tactile channel remains relatively underexplored [ 11 ] . This neglect is partly due to the interdisciplinary challenges inherent in studying affective touch [ 4 ] . However, prior studies have shown that tactile interaction with robots can significantly influence human behaviours, such as risk-taking, stress responses, and attitudes toward the robot [ 12 ] . Recent research has increasingly explored the role of vibration and sound in emotion expression, demonstrating that vibrotactile signals can effectively convey emotions such as happiness, sadness, and anger [ 13 ] . Specific vibration patterns, defined by variations in frequency, amplitude, and rhythm, have been mapped to distinct emotional states, contributing to affective perception [ 14 ] . Notably, touch is rarely experienced in isolation; it is inherently multisensory [ 15 ] . When interacting with vibrotactile systems, individuals often experience not only the tactile sensations but also the accompanying auditory cues produced by touch. Recent research, for instance, has investigated how people decode emotional intent and specific touch gestures from these tactile sounds. [ 16 ] . This co-occurrence of tactile and auditory feedback has important implications for affective human-robot interaction (HRI). While computer vision has been widely adopted for emotion recognition, alternative modalities become essential in contexts where visual input is limited or unreliable [ 17 ] . Auditory signals, based on their temporal and spectral properties, can independently convey affective qualities such as urgency, warmth, or discomfort [ 18 ] . Moreover, previous studies further indicate that tactile gestures and emotional expressions can be recognized through their auditory counterparts alone [ 19 ] . Despite these insights, the potential of integrating tactile and auditory modality during human-robot tactile interaction remains underexplored. Most existing studies focus solely on vision and sound [ 20 ] for emotion expression [ 21 ] , overlooking the communicative richness of multimodal tactile–auditory signals. There is limited understanding of how humans perceive and decode both the combination of touch and touch-generated sound, as well as each modality’s contribution in affective and social contexts in HRI. Addressing this gap is crucial for developing emotionally intelligent robots capable of effective communication when visual cues are limited, and enabling remote interactions with both human partners and virtual agents. One key challenge in HRI is the physical limitation of current humanoid platforms. For example, robots such as Pepper and Nao offer only limited tactile expressivity, as their grippers are typically restricted to basic open–close motions. In contrast, advances in mediated touch technologies—such as haptic gloves, vibration devices, and smart bands—have demonstrated the effectiveness of tactile feedback in virtual and augmented reality, highlighting the potential for richer multisensory integration in HRI [ 22 , 23 ] . However, these devices are rarely designed for emotional communication in HRI, and their integration with other sensory modalities remains limited. Furthermore, the perception and interpretation of affective touch can be influenced by the embodiment of the humanoid robot, making it valuable to investigate how such technologies might be adapted for human–robot interaction. To address this, we employ mediated tactile technologies, specifically, wearable vibrotactile devices that enable spatially distributed and dynamic feedback. In this context, we developed the VibroSleeve, a wearable haptic interface that a humanoid robot delivers spatially distributed vibration to the users to convey different emotions. By integrating vibrotactile and auditory modalities, this study proposes a multimodal framework for conveying affective and touch gestures from a humanoid robot to humans and investigates the following research questions: 1. How do people interpret emotions and touch gestures conveyed through auditory and haptic modalities in human–robot interaction? Does the combination of haptic and auditory cues generated by touch enhance the accuracy of emotion and touch gesture recognition compared to either modality alone? 2. Do participants associate certain touch gestures with specific emotional meanings, even when those gestures were not explicitly designed to convey particular affective responses? 3. What are the individual contributions of the haptic and auditory channels to emotion and touch gestures decoding, and how does their effectiveness vary across different emotional expressions? 4. Which emotions

Read the complete text