From Multimodal Signals to Adaptive XR Experiences for De-escalation Training

Birgit Nierula, Karam Tomotaki-Dawoud, Daniel Johannes Meyer, Iryna Ignatieva, Mina Mottahedin, Thomas Koch, Sebastian Bosse · 2026

Read From Multimodal Signals to Adaptive XR Experiences for De-escalation Training by Birgit Nierula, Karam Tomotaki-Dawoud, Daniel Johannes Meyer, Iryna Ignatieva, Mina Mottahedin, Thomas Koch, Sebastian Bosse (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11570.txt.

Introduction

Nierula and six colleagues present an early-stage multimodal system for XR de-escalation training. Five synced streams cover speech and prosody, body gestures from multi-view cameras, and affective signals from face video and facial EMG. EEG mental-state decoding and arousal from skin conductance, heart activity, and spacing complete the set. An interpretation layer maps these signals to escalation and easing tension, informed by police trainers and lay volunteers. Early results show feasibility and limits of automated cue reading under headset blocking.

Catalog record

CREATORBirgit Nierula, Karam Tomotaki-Dawoud, Daniel Johannes Meyer, Iryna Ignatieva, Mina Mottahedin, Thomas Koch, Sebastian Bosse
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44100
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.11570

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Multimodal Signals to Adaptive XR Experiences for De-escalation Training Birgit Nierula, Karam Tomotaki-Dawoud, Daniel Johannes Meyer, Iryna Ignatieva, Mina Mottahedin, Thomas Koch, Sebastian Bosse Abstract We present the early-stage design and implementation of a multimodal, real-time communication analysis system intended as a foundational interaction layer for adaptive VR training. The system integrates five parallel processing streams: (1) verbal and prosodic speech analysis, (2) skeletal gesture recognition from multi-view RGB cameras, (3) multimodal affective analysis combining lower-face video with upper-face facial EMG, (4) EEG-based mental state decoding, and (5) physiological arousal estimation from skin conductance, heart activity, and proxemic behavior. All signals are synchronized via Lab Streaming Layer to enable temporally aligned, continuous assessments of users’ conscious and unconscious communication cues. Building on concepts from social semiotics and symbolic interactionism, we introduce an interpretation layer that links low-level signal representations to interactional constructs such as escalation and de-escalation. This layer is informed by domain knowledge from police instructors and lay participants, grounding system responses in realistic conflict scenarios. We demonstrate the feasibility and limitations of automated cue extraction in an XR-based de-escalation training project for law enforcement, reporting preliminary results for gesture recognition, emotion recognition under HMD occlusion, verbal assessment, mental state decoding, and physiological arousal. Our findings highlight the value of multi-view sensing and multimodal fusion for overcoming occlusion and viewpoint challenges, while underscoring that fusion and feedback must be treated as design problems rather than purely technical ones. The work contributes design resources and empirical insights for shaping human–AI-powered XR training in complex interpersonal settings. [orcid=0000-0003-2174-5872, [email protected], ] *1 1 [orcid=0009-0006-2745-8312, ] 1 [orcid=0009-0009-6039-7234, ] 1 keywords Multimodal fusion ,Nonverbal cue recognition ,Immersive virtual reality ,XR-based training ,Physiological signal analysis † † copyrightyear: 2026 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † address: Department for Vision and Imaging Technologies, Fraunhofer Heinrich-Hertz-Institute, Einsteinufer 37, 10587 Berlin, Germany † † corresp: Corresponding author. † † These authors contributed equally. 1 Introduction Human communication is inherently multimodal, combining verbal content with non-verbal signals such as body posture, gestures, facial expressions, vocal prosody, and interpersonal space 26 . In emotionally charged interactions, these non-verbal cues often carry as much -or more- meaning than spoken language 46 . They are frequently produced unconsciously 31 , yet they strongly shape how intentions are perceived and whether an interaction escalates or de-escalates 46 . For professionals operating in high-stakes contexts, such as law enforcement 51 or healthcare 53 , the ability to regulate both verbal and non-verbal communication is therefore a critical skill. Traditional communication training methods 64 ; 54 ; 14 , including role-play and instructor-led feedback, can support the development of these skills but are limited by subjectivity, delayed feedback, and the difficulty of making unconscious communication patterns visible to trainees. Video-based review and post-hoc analytics offer additional insight, but they separate reflection from the lived, embodied experience of the interaction itself. Immersive virtual reality (VR) provides a promising medium for communication training by enabling embodied, situated interactions that evoke realistic emotional and behavioural responses while remaining safe, repeatable, and controllable 52 . Prior studies have shown that VR can elicit affective and physiological reactions comparable to real-world encounters 61 ; 33 . However, designing VR systems for interpersonal skill training introduces a fundamental challenge: communication unfolds continuously across multiple modalities, and its meaning is highly context-dependent 21 . Systems that rely on predefined choices, scripted branching, or post-session evaluation therefore struggle to capture the dynamic nature of real-world interactions. To shape meaningful training experiences, XR environments must be able to sense and interpret users’ communicative behaviour in real time and translate it into adaptive responses that remain immersive and interpretable. This raises a central design question for XR: how can multimodal communication signals be operationalized in a way that supports adaptive, experience-driven training without disrupting the interaction itself? Addressing this question requires not only accurate signal processing, but also careful design decisions regarding which cues matter, how they are interpreted, and how they influence virtual character behaviour and feedback. Research Gap. Limitations of current solutions are their lack of objective feedback and scalability as they are typically implemented as role-play and instructor-led feedback. Objective and scalable XR solutions that allow to capture and predict the dynamics of social interactions are challenged by a lack of multimodal (verbal and non-verbal) communication sensing in real-time in a context-aware manner. This, however, is necessary for adaptive, context-aware avatar responses that combined with objective and transparent feedback can foster awareness and control over own verbal and non-verbal communication cues. Implementing this requires both technical signal processing capabilities and social theory design frameworks for operationalizing communication cues. Contributions. This work offers the following contributions to the design of multimodal communication sensing XR systems grounded in social science: • Implemented and evaluated: – A multi-view skeletal gesture recognition pipeline achieving 82–88% accuracy for conflict-relevant postures and gestures; – A late-fusion architecture combining lower-face video with upper-face EMG for emotion recognition under HMD occlusion, achieving 51% macro-F1 on 7-class classification; – Physiological arousal analysis (SCR, HRV) validated in a controlled experimental scenario with sensitivity to proxemic violations and emotional facial expressions; – Verbal communication analysis including formality detection (68% accuracy) – prosodic feature extraction with preliminary correlation analysis. • Implemented with preliminary evaluation: – EEG-based mental state decoding using SSD+SPoC decomposition, demonstrated as proof-of-concept in 3 participants; – Sentence complexity assessment, evaluated on external benchmark data but not yet validated in police interaction contexts. • Conceptual contributions: – An integrated system architecture synchronizing five parallel analysis streams via Lab Streaming Layer for temporally aligned multimodal assessment; – A design framework grounded in social semiotics and symbolic interactionism, linking low-level signal representations to interactional constructs (escalation/de-escalation) through domain expert and lay participant input; – Design considerations for multimodal fusion and adaptive feedback in XR training, framed as open design problems rather than solved technical challenges. By distinguishing evaluated components from preliminary and conceptual contributions, we aim to provide transparency about the current maturity of each system element. We illustrate this approach through its application in an ongoing research project on de-escalation training for law enforcement. We report preliminary findings and design reflections that suggest the feasibility of using multimodal communication analysis to inform real-time adaptation in immersive environments. By foregrounding design considerations and early insights, this work contributes to the ShapeXR community by highlighting how human–AI-powered XR experiences can be used to support embodied learning in complex interpersonal scenarios. 2 Related Work In recent years, a range of product developments and research activities have sought to exploit the advantages of XR-based training in the policing context. The TARGET innovation project 58 developed a toolkit for creating integrated AR/VR training scenarios, which also enables the representation of complex, pan-European large-scale operations. Similarly, many existing training solutions focus on scenario simulation to train procedures, behavioural patterns, decision-making, and communication from a tactical, process-oriented perspective 49 ; 50 ; 55 ; 2 . These approaches often incorporate haptic elements through the use of operational equipment, particularly weapons 45 . Data-driven tactical and behavioural analyses are typically conducted after the training, based on video data, for example 19

Read the complete text