REVELIO - Universal Multimodal Task Load Estimation for Cross-Domain Generalization

Maximilian P. Oppelt, Andreas Foltyn, Nadine R. Lang-Richter, Bjoern M. Eskofier · 2025

Read REVELIO - Universal Multimodal Task Load Estimation for Cross-Domain Generalization by Maximilian P. Oppelt, Andreas Foltyn, Nadine R. Lang-Richter, Bjoern M. Eskofier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01642.txt.

Introduction

Estimating how loaded a person's mind is during a task helps tune work and training, but models trained in one setting often fail in another. Maximilian P. Oppelt, Andreas Foltyn, Nadine R. Lang-Richter, and Bjoern M. Eskofier introduce REVELIO, a multimodal data set and evaluation for universal task load estimation. It extends established cognitive load benchmarks with a real-world gaming application, grounded in the classic n-back memory test.

Task load labels come from objective performance, subjective workload ratings on a standard scale, and the task design itself. Leading end-to-end models, including recurrent, convolutional, and transformer designs, were trained and tested across several data types and domains. Multimodal approaches consistently beat single-source baselines, though which sources and designs helped most varied by application. The team aims at load models that travel across real settings.

Catalog record

CREATORMaximilian P. Oppelt, Andreas Foltyn, Nadine R. Lang-Richter, Bjoern M. Eskofier
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-01642

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

REVELIO - Universal Multimodal Task Load Estimation for Cross-Domain Generalization Maximilian P. Oppelt, Andreas Foltyn, Nadine R. Lang-Richter, Bjoern M. Eskofier Abstract Task load detection is essential for optimizing human performance across diverse applications, yet current models often lack generalizability beyond narrow experimental domains. While prior research has focused on individual tasks and limited modalities, there remains a gap in evaluating model robustness and transferability in real-world scenarios. This paper addresses these limitations by introducing a new multimodal dataset that extends established cognitive load detection benchmarks with a real-world gaming application, using the n n -back test as a scientific foundation. Task load annotations are derived from objective performance, subjective NASA-TLX ratings, and task-level design, enabling a comprehensive evaluation framework. State-of-the-art end-to-end model, including xLSTM, ConvNeXt, and Transformer architectures are systematically trained and evaluated on multiple modalities and application domains to assess their predictive performance and cross-domain generalization. Results demonstrate that multimodal approaches consistently outperform unimodal baselines, with specific modalities and model architectures showing varying impact depending on the application subset. Importantly, models trained on one domain exhibit reduced performance when transferred to novel applications, underscoring remaining challenges for universal cognitive load estimation. These findings provide robust baselines and actionable insights for developing more generalizable cognitive load detection systems, advancing both research and practical implementation in human-computer interaction and adaptive systems. I Introduction Managing mental workload experienced by humans typically aims to achieve the objectives of optimizing the subject’s performance in a specific task, enhancing engagement or reducing frustration, and minimizing errors or accidents. To realize these goals, it is crucial to measure mental workload to quantify the mental cost of performing a task. However, the measurement is either very domain-specific or relies on a single technique to assess mental workload [ 1 ] . Similarly, scholars have developed a theoretical foundation in psychology characterizing mental workload as a complex construct that is dynamic, nonlinear, person-specific, and multidimensional and closely linked to attention and effort. However, each theory presents different perspectives that are intertwined and cannot be analyzed independently when working with the construct of mental workload. Cognitive Load Theory by Sweller [ 2 ] , for instance, identifies intrinsic task performance alongside extraneous and germane load, recognizing task complexity as a significant factor. In contrast, Flow Theory [ 3 ] focuses on the perceived demand as a critical characteristic. Arousal Theory [ 4 ] emphasizes maintaining arousal at an optimal moderate level to ensure peak performance and Multiple Resource Theory [ 5 ] highlights the constraints of limited attentional resources and the finite nature of attention and cognition. An extensive analysis of these theoretical frameworks show that they complement each other and capture different aspects of mental workload [ 1 ] . In addition to these theoretical constructs, practitioners studying mental workload in experimental settings typically employ multiple modalities utilizing measures from four distinct groups: self-reported subjective measures, such as questionnaires that quantitatively report personal experiences; behavioral measures , including gaze or certain facial expressions; physiological measures , such as changes in metrics like heart rate, heart rate variability, and respiration, as well as pupil changes; and performance measures , such as hit rate or reaction times [ 6 ] . These measurements have been used to explore cognitive load in various contexts. For instance, tests like the n -back test are meticulously designed to induce cognitive load. Additionally, practitioners conducted application tests aiming at enhancing the safety in settings such as in driving or boosting the long-term user engagement or learning efficiency in video games, that are crafted to optimize cognitive load. Although these applications all focus cognitive load, each tends to emphasize distinct facets of this inherently multidimensional construct. As a result, detection models developed for one context are often limited in their applicability and cannot be transferred to other domains. One key limitation stems from inconsistencies in the types of input data collected across different settings. For example, in video game environments, cognitive load can be assessed consistently without the influence of physical movement or changes in the surrounding environment. By contrast, evaluating cognitive load in lower limb prosthesis users during ambulatory activities necessitates capturing data during diverse physical actions such as sitting, walking, and running across a range of environments [ 7 ] . Furthermore, participant characteristics, such as skill and experience, may serve as confounding variables. In driving studies, for instance, selecting drivers based on years of holding a driver’s license, or in video game research, by an estimated number of hours of play, can significantly impact task load levels and induced mental workload. Additionally, experimental protocols may introduce further confounds, such as the induction of emotions with varying valence and arousal, all of which can influence the measurement and interpretation of cognitive load. Given the multidimensional and context-dependent nature of cognitive load, this study is guided by the following research questions: First, to what extent can commonly used, continuous measurements, accurately predict task load during levels designed to impose varying levels of mental demand, as validated by self-reported subjective and performance metrics ( RQ1 )? Second, how effectively do task load detection systems generalize from one application domain represented in the training data to other, distinct application scenarios ( RQ2 )? To address these questions, we take a step toward a more universally applicable task load detection system with our proposed framework, R obust E stimation V ia E nd-To-End L earning of Multimodal O bservations (REVELIO) . The framework introduces two new practical application scenarios for cognitive load detection, comprising recordings from two widely played video games, Overcooked! 2 and Hogwarts Legacy , alongside an n -back task. We extend existing data obtained from a driving simulator study, thereby ensuring consistency in measurement modalities, recording equipment, and n -back test protocols. This foundation enables us to build the framework’s evaluation protocol, in which we first train and then evaluate state-of-the-art classification architectures using an end-to-end learning paradigm. This approach obviates labor-intensive feature engineering and subject-specific normalization and is assessed across multiple scenarios and input modalities. Finally, we systematically investigate cross-domain robustness by evaluating systems trained on one task and tested on another, thereby providing a principled basis for assessing model performance and behavior under domain transfer. II Related Work TABLE I: Empirical studies and datasets for cognitive load detection in various contexts with different stimuli and underlying psychological concepts and their respective measurements and recording setups. Reference Physiological Subjective Behavioral Performance Subjects Setup Stimulus Concept [ 8 ] ECG, EDA, RESP NASA-TLX - non-driving related task performance 49F, 40M, 1 other driving simulator driving & verbal backward counting cognitive load [ 9 , 10 ] EEG, ET, EDA, RSP, fNIRS, BP, SpO2 - ET stimulus onset asynchrony 36 \dagger F, 46M driving simulator questions and braking events cognitive load, distraction, mind wandering [ 11 ] ACC, EDA, TEMP, PPG NASA-TLX - time on task, number often answers, game points 46 (23 per task) lab standardized tests ( n -back), video game cognitive load [ 12 , 13 ] EEG, ECG, EDA, ET NASA-TLX ET mean and standard deviation of speed 15F, 18M driving simulator driving tasks and modified n -back cognitive load [ 14 ] ECG, EDA, PPG NASA-TLX - invalid responses,

Read the complete text