Hand by Hand: LLM Driving EMS Assistant for Operational Skill Learning

Wei Xiang, Ziyue Lei, Haoyuan Che, Fangyuan Ye, Xueting Wu, Lingyun Sun · 2025

Read Hand by Hand: LLM Driving EMS Assistant for Operational Skill Learning by Wei Xiang, Ziyue Lei, Haoyuan Che, Fangyuan Ye, Xueting Wu, Lingyun Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06000.txt.

Introduction

Wei Xiang and co-authors let a language model move a trainee's arm to teach a hands-on skill. Skills like flying are learned through the body, yet model-based tutors mostly send text, missing the feel of a correct motion. Their 2025 work pairs the model with electrical muscle stimulation, which nudges muscles directly, using simulated flight as the test case.

Their tool, FlightAxis, reads flight manuals and guides the forearm during practice runs, following an align, analyze, adjust loop. Trainees accepted the model-driven control well and finished tasks much faster. Rather than feeling carried, they said the physical guidance made their errors easier to notice and kept them more engaged. The authors read this as proof that language models can teach through touch, not just words, opening a path for training in other manual trades.

Catalog record

CREATORWei Xiang, Ziyue Lei, Haoyuan Che, Fangyuan Ye, Xueting Wu, Lingyun Sun
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-06000

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hand by Hand: LLM Driving EMS Assistant for Operational Skill Learning Wei Xiang, Ziyue Lei, Haoyuan Che, Fangyuan Ye, Xueting Wu, Lingyun Sun Abstract Operational skill learning, inherently physical and reliant on hands-on practice and kinesthetic feedback, has yet to be effectively replicated in large language model (LLM)-supported training. Current LLM training assistants primarily generate customized textual feedback, neglecting the crucial kinesthetic modality. This gap derives from the textual and uncertain nature of LLMs, compounded by concerns on user acceptance of LLM driven body control. To bridge this gap and realize the potential of collaborative human-LLM action, this work explores human experience of LLM driven kinesthetic assistance. Specifically, we introduced an “Align-Analyze-Adjust” strategy and developed FlightAxis, a tool that integrates LLM with Electrical Muscle Stimulation (EMS) for flight skill acquisition, a representative operational skill domain. FlightAxis learns flight skills from manuals and guides forearm movements during simulated flight tasks. Our results demonstrate high user acceptance of LLM-mediated body control and significantly reduced task completion times. Crucially, trainees reported that this kinesthetic assistance enhanced their awareness of operation flaws and fostered increased engagement in the training process, rather than relieving perceived load. This work demonstrated the potential of kinesthetic LLM training in operational skill acquisition. 1 Introduction Operational skills such as driving, riding, and operating surgeries occupied diverse aspects of life and work. Mastering operational skills requires not only theoretical knowledge but also procedure precision and operation stability, ensuring critical metrics within acceptable ranges García-Jiménez et al. 2015 . While traditional training methods, such as self-directed reflection and expert instruction, remain valuable, Large Language Model (LLM)-supported assistance has recently emerged as a promising and rapidly expanding area of research. LLM acquires expertise and experience from extensive literature, and possesses the ability to analyze multi-modal training data, positioning them to provide tailored feedback aligned with trainees’ progress Yu et al. 2024 . Furthermore, LLM-supported systems offer personalized guidance and professional responses, supporting trainees to practice at an individualized pace Xu et al. 2024 . Figure 1: The main contribution of our work to exploring LLM-assisted skill learning with kinesthetic feedback Despite the strengths, a critical gap remains. In accordance with the language base of LLM, LLM-based systems employ the mainstream way of human-LLM collaboration and assist trainees through vocal and textual feedback Du et al. 2024 ; Zhao et al. 2024 . This is natural for LLM outputs while leaving the kinesthetic feedback underexplored. These operational skills are inherently physical, requiring hands-on practice and the development of muscle memory through kinesthetic feedback, both for novice and experts Furuya et al. 2025 . However, this transformation from vocal to tactile is not straightforward. Vocal instructions, such as “push forward”, have ambiguities in amplitude and velocity compared to kinesthetic instructions. Also, body control by LLM may lead to resistance even in the context of operational skill learning. How would trainees perform with and react to this kinesthetic feedback? This is the key research question of our article. This work focuses on flight as a representative operational skill domain to explore human interaction with LLM-driven kinesthetic assistance. Specifically, we developed FlightAxis 1 1 1 Supplementary Materials: https://github.com/Z500-RAY/LLM-Driving-EMS-Assistant , a LLM-supported prototype that utilizes Electrical Muscle Stimulation (EMS) to guide the forearm movement through flight tasks. EMS, by inducing involuntary muscle contractions through targeted electrical stimulation, provides a direct and tangible form of kinesthetic feedback. FlightAxis leverages an LLM trained in flight manuals and expert knowledge to generate personalized guidance, translating textual instructions into EMS patterns that assist trainees in executing flight maneuvers. To the best of our knowledge, it is the first study that combines LLM and EMS for operational skill learning. A comparative study 2 2 2 Ethical approval: Organization Ethical Review Committee was conducted to the effectiveness and user experience of LLM-driven kinesthetic feedback. Participants exhibited positive reactions but also express concerns on specific scenarios, providing insights for design of kinesthetic assistance systems. Overall, the contributions include: 1. This study explores the integration of LLM in operational skills learning. LLM and EMS reinforce the learning environment of embodied cognition, making the learning process dynamic and interactive. 2. This study developed a flight operation skill learning prototype. It employs LLM to align relevant operations, analyze flight status, and generate targeted EMS patterns to adjust arm movements, thereby supporting flight skill learning. 3. This study explored human behavior and feelings towards kinesthetic feedback through an empirical study. Our results demonstrate that this approach not only improves operational performance but also enhances trainees’ awareness of their own skill gaps. These findings offer valuable insights for designing effective and user-centered kinesthetic training systems. 2 Related Work 2.1 LLM-Assisted Skill Learning The application of LLM in skill learning has a solid foundation. Compared to textual manual and recorded video instructions, LLM summarizes operation instructions and enables interactive conversation. This interaction make LLMs ideal for skill learning assistance. For example, Murtaza et al. demonstrated that the use of a ChatGPT-Based Interactive User Manual could improve training outcomes by constructing an flexible learning environment. The trainees could engage in a textual Q&A with a context-aware manual pre-configured with training scenarios on a simulator screen. Their results showed this approach reduced reaction time for ADAS function activation and increased accuracy Murtaza et al. 2024 . Also, Varas et al. reported LLM transcription and summarization of communication among trainees and instructors, thus improving learning outcomes Varas et al. 2023 . The second advantage of LLM lies in its ability to provide personalized guidance. LLMs monitor trainees’ performance and remind criterion whenever needed. For instance, Kabir et al. proposes an online learning system where LLMs, like ChatGPT, generate personalized practice questions, answers, and feedback based on trainees’ knowledge levels, offering a more tailored learning experience Kabir and Lin 2023 . Also, Looi et al. designed a framework of personalized learning affordance enabled by ChatGPT where the tutoring chatbot delivers responses at multiple personalization levels through corresponding prompts Looi and Jia 2025 . The chatbot effectively identified learners’ individual learning gaps and provided personalized support to address them, showcasing LLMs’ ability as a training companion. In general, LLMs have shown the ability to summarize manual of operational skills and give instructions. Inherited by its textual nature, most LLM-supported systems outputs vocal or visual feedback, and have not provide hands-on instructions and kinesthetic feedback for operational skills. Differed from vocal feedback, this direct manipulation of human body meets the nature of physical training, while also requires accurate, short and acceptable outputs. 2.2 Skill Learning From Kinesthetic Feedback Kinesthetic feedback is a widely accepted and essential mode of operational skill learning. Kinesthetic feedback makes full use of human receptors in muscles, skin, tendons, and joints to remind trainees of position, velocity, and force of operations Schostek et al. 2009 . This feedback helps human locate body parts in space and master the dynamics of movement, even without visual information Pinzon et al. 2017 . Because operational skills involve body movements and emphasize procedural knowledge, kinesthetic feedback helps trainees refine their actions. For example, providing force and position guidance from robot arm in welding reduced novice welders’ task completion time and improved accuracy Ye et al. 2024 . Similarly, multi-directional force feedback and tool-tip tactile sensation significantly

Read the complete text