GestureCoach : Rehearsing for Engaging Talks with LLM-Driven Gesture Recommendations

Ashwin Ram, Varsha Suresh, Artin Saberpour Abadian, Vera Demberg, Jürgen Steimle · 2025

Read GestureCoach : Rehearsing for Engaging Talks with LLM-Driven Gesture Recommendations by Ashwin Ram, Varsha Suresh, Artin Saberpour Abadian, Vera Demberg, Jürgen Steimle (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10706.txt.

Introduction

Ram and colleagues introduce in 2025 GestureCoach, a rehearsal system that teaches speakers to gesture well using language model guidance. It was trained on how expert TED speakers move. One module spots note passages worth a gesture. Another picks a fitting gesture from a curated library. In tests, the spotter beat general-purpose models at choosing gesture moments.

Speakers rated most suggested moments and gestures as apt. Rehearsing with the system made people gesture more and with more variety, and talks felt more engaging. The authors draw lessons for future AI rehearsal tools.

Catalog record

CREATORAshwin Ram, Varsha Suresh, Artin Saberpour Abadian, Vera Demberg, Jürgen Steimle
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-10706

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GestureCoach : Rehearsing for Engaging Talks with LLM-Driven Gesture Recommendations Ashwin Ram, Varsha Suresh, Artin Saberpour Abadian, Vera Demberg, Jürgen Steimle Abstract . This paper introduces GestureCoach , a system designed to help speakers deliver more engaging talks by guiding them to gesture effectively during rehearsal. GestureCoach combines an LLM-driven gesture recommendation model with a rehearsal interface that proactively cues speakers to gesture appropriately. Trained on experts’ gesturing patterns from TED talks, the model consists of two modules: an emphasis proposal module, which predicts when to gesture by identifying gesture-worthy text segments in the presenter notes, and a gesture identification module, which determines what gesture to use by retrieving semantically appropriate gestures from a curated gesture database. Results of a model performance evaluation and user study (N=30) show that the emphasis proposal module outperforms off-the-shelf LLMs in identifying suitable gesture regions, and that participants rated the majority of these predicted regions and their corresponding gestures as highly appropriate. A subsequent user study (N=10) showed that rehearsing with GestureCoach encouraged speakers to gesture and significantly increased gesture diversity, resulting in more engaging talks. We conclude with design implications for future AI-driven rehearsal systems. Figure 1. GestureCoach guides speakers to perform gestures while rehearsing their talk. A gesture recommendation model predicts text segments in the presenter notes that should be emphasized with gestures and retrieves relevant semantic gestures for each segment. During rehearsal, the system highlights the segments and proactively cues a video clip of the gesture by tracking users’ speech, allowing them to integrate the gesture smoothly into their talk. Hovering over a segment allows users to preview and modify the associated gesture with alternate suggestions from the model. The left side of the figure shows a speaker rehearsing for a presentation talk with the GestureCoach system. The speaker is saying "one key reason" and showing the gesture symbolizing one by following a gesture cue provided by the system where a humanoid avatar demonstrates the same gesture. The right side shows a zoomed in view of the rehearsal interface. The interface has the appeal of the presenter view mode typically found in presentation software. It has 4 main windows. 1) the current slide 2) the presenter notes for the slide with text segments where the speaker should perform gestures highlighted in yellow. This window also has a rehearse button that speakers click to start rehearsal. 3) a window where a humanoid avatar displays the gesture that should be performed for each highlighted region. 4) a popup window which appears when speakers hover over a highlighted region. This window shows video clips of the top-3 gesture recommendations provided by the recommendation model as a carousel design. It has an edit and delete button that allows users to browse and select the gesture they prefer for a region or remove the region if they do not prefer to gesture there. 1. Introduction Delivering a “great” talk requires more than just good content; it needs an engaging delivery with the right blend of non-verbal cues, such as gestures ( Goldin-Meadow and Alibali 2013 ; Krauss et al. 1996 ; McNeill 2008 ) . In particular, semantic gestures that complement the spoken content ( McNeill 1992 ) , such as tracing the outline of a circle in the air while saying ’a round table’ or moving hands upward while saying ’rising prices’, play a critical role in enhancing the dynamism of the speech ( BURGOON et al. 1990 ; Rodero 2022 ; Kendon 1994 ) and facilitating comprehension ( Maricchiolo et al. 2020 ; Kendon 1994 ; Hostetter 2011 ) . However, speakers, especially novices, often struggle with using semantic gestures in their talks ( So et al. 2013 ; Zeng et al. 2022 ) . They may either completely neglect to use gestures or rely heavily on beat gestures ( McNeill 1992 ) —simple, repetitive hand movements that do not enhance the meaning of the content. While some existing training systems ( Schneider et al. 2015 ; Damian et al. 2015 ; Barmaki 2016 ; Tanveer et al. 2015 ) track users’ motion and provide generic feedback to use more or fewer gestures, they do not offer specific guidance on what semantic gesture to use or when to use it for effective emphasis. Prior research has shown that verbal qualities, such as voice modulation, can be improved with rehearsal and then effectively applied during live talks ( Wang et al. 2020 ) . Inspired by this work, we aim to develop a rehearsal system specifically designed to improve semantic gesture usage during talks. Designing a rehearsal system that can give targeted guidance for semantic gestures, however, presents various technical and design challenges. The key difficulty lies in developing a gesture recommendation model that predicts appropriate text segments in the presenter notes that should be emphasized with gestures (henceforth referred to as gesture regions) and retrieves suitable semantic gestures for effective emphasis. Predicting whether a text segment is a gesture region depends on several factors, including its broader context within the text, the topic of the talk, and implicit knowledge expert speakers use to engage an audience. Furthermore, the suggested gestures must extend beyond the limited gesture vocabulary of novices, offering access to the more refined and expressive gestures used by experts. From a design perspective, these gesture recommendations must be delivered proactively during rehearsal, allowing users to practice and seamlessly incorporate them into their talk. As a step towards tackling these challenges, we propose GestureCoach , the first system capable of providing targeted guidance to perform gestures during rehearsal. GestureCoach combines a gesture recommendation system with a rehearsal interface that delivers real-time gesture cues, helping speakers naturally integrate gestures while speaking. The gesture recommendation system utilizes recent advances in AI chaining, by combining two specialized Large Language Model (LLM) modules, each targeting a specific aspect of gesture-based emphasis. First, the emphasis proposal module takes presenter notes as input and determines appropriate gesture regions. For this, we leverage the foundational knowledge of LLMs and further fine-tune it with annotated data from 10 TED talk videos of expert speakers. Second, the gesture identification module selects the specific semantic gesture to emphasize each predicted gesture region, by retrieving relevant examples from a curated database of gestures performed by experts using a Retrieval-Augmented Generation (RAG) framework ( Lewis et al. 2020 ) . We then identified key design requirements for a gesture rehearsal system through an iterative design process, which included developing and evaluating an early Wizard-of-Oz prototype in a pilot study with five participants. This study highlighted the need for an effective Human-AI collaborative interface design; one that enables users to proactively engage with and co-create the AI’s gesture recommendations. Based on these findings, we developed a desktop-based rehearsal interface (shown in Figure 1 ) that tracks users’ speech in real-time and proactively cues users during their practice as and when gestures need to be performed. Additionally, the system supports collaboration by allowing users to modify the AI’s recommendations, enabling the creation of personalized gesture palettes tailored to each speaker’s preferences. We evaluate our system in two ways: 1) technical evaluation of our gesture recommendation model, and 2) user evaluation of the final GestureCoach system. Results show

Read the complete text