A Platform-Agnostic Multimodal Digital Human Modelling Framework: Neurophysiological Sensing in Game-Based Interaction

Daniel J. Buxton, Mufti Mahmud, Jordan J. Bird, Thomas Hughes-Roberts, David J. Brown · 2026

Read A Platform-Agnostic Multimodal Digital Human Modelling Framework: Neurophysiological Sensing in Game-Based Interaction by Daniel J. Buxton, Mufti Mahmud, Jordan J. Bird, Thomas Hughes-Roberts, David J. Brown (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10680.txt.

Introduction

Daniel J Buxton and co-authors present a platform-agnostic framework for digital human modelling with neurophysiological sensing in game-based interaction. Many such setups are tied to one platform or pipeline, limiting reuse. This work separates sensing, interaction modelling, and inference readiness, giving a base other groups can reuse. An OpenBCI Galea headset supplies EEG, EMG, EOG, PPG, and inertial streams at once, while a SuperTux game supplies the interaction. Signals are stored as time-aligned observables for later AI use under ethical approval.

Catalog record

CREATORDaniel J. Buxton, Mufti Mahmud, Jordan J. Bird, Thomas Hughes-Roberts, David J. Brown
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19830
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.10680

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Platform-Agnostic Multimodal Digital Human Modelling Framework: Neurophysiological Sensing in Game-Based Interaction Daniel J. Buxton, Mufti Mahmud, Jordan J. Bird, Thomas Hughes-Roberts, David J. Brown Abstract Digital Human Modelling (DHM) is increasingly shaped by advances in artificial intelligence (AI), wearable biosensing, and interactive digital environments, particularly in research addressing accessibility and inclusion. However, many AI-enabled DHM approaches remain tightly coupled to specific platforms, tasks, or interpretative pipelines, limiting reproducibility, scalability, and ethical reuse. This paper presents a platform-agnostic DHM framework designed to support AI-ready multimodal interaction research by explicitly separating sensing, interaction modelling, and inference readiness. The framework integrates the OpenBCI Galea headset as a unified multimodal sensing layer, providing concurrent Electroencephalogram (EEG), Electromyogram (EMG), Electro-oculogram (EOG), Photoplethysmogram (PPG), and inertial data streams, alongside a reproducible, game-based interaction environment implemented using SuperTux. Rather than embedding AI models or behavioural inference, physiological signals are represented as structured, temporally aligned observables, enabling downstream AI methods to be applied under appropriate ethical approval. Interaction is modelled using computational task primitives and timestamped event markers, supporting consistent alignment across heterogeneous sensors and platforms. Technical verification via author self-instrumentation confirms data integrity, stream continuity, and synchronisation; no human-subjects evaluation or AI inference is reported. Scalability considerations are discussed with respect to data throughput, latency, and extension to additional sensors or interaction modalities. Illustrative use cases demonstrate how the framework can support AI-enabled DHM and HCI studies, including accessibility-oriented interaction design and adaptive systems research, without requiring architectural modifications. The proposed framework provides an emerging-technology-focused infrastructure for future ethics-approved, inclusive DHM research. 1 Introduction Digital Human Modelling (DHM) plays a central role in the design of human–computer systems across domains such as ergonomics, safety, health, and accessibility. Recent advances in wearable sensing and interactive technologies have expanded the range of signals available for modelling human interaction, including neurophysiological, muscular, ocular, and cardiovascular measures. At the same time, there is growing recognition that accessibility and inclusion must be treated as first-class design considerations within DHM, particularly when research aims to support diverse populations and contexts. Despite these advances, many existing digital modelling and multimodal interaction approaches remain tightly coupled to specific platforms, experimental setups, or task environments. Sensing, interaction, and interpretation are often integrated within bespoke pipelines optimised for a single study or application, limiting reproducibility, portability, and ethical reuse. This coupling presents challenges for accessibility-oriented research, where interaction tasks and sensing configurations may need to be adapted to accommodate differing motor, sensory, or cognitive needs without re-engineering the entire system. In parallel, the use of neurophysiological signals in human–computer interaction has raised important ethical considerations. While such signals can provide valuable contextual information about interaction, their interpretation is frequently conflated with inference about internal cognitive or emotional states. For DHM research, particularly in accessibility-sensitive contexts, there is a need for infrastructures that clearly separate data acquisition from interpretation, allowing physiological and interaction data to be treated as descriptive observables rather than diagnostic indicators. This paper addresses these challenges by presenting a platform-agnostic multimodal DHM framework that decouples neurophysiological sensing, interaction modelling, and inference readiness through a modular abstraction architecture. The framework integrates the OpenBCI Galea headset as a unified sensing layer, providing concurrent neurophysiological and inertial data streams, alongside a reproducible, game-based interaction environment implemented using SuperTux. Interaction is modelled through structured task primitives and timestamped event markers, enabling consistent alignment between sensing and interaction while remaining independent of specific hardware or software platforms. The contribution of this work is architectural rather than evaluative. Technical verification is limited to the authors’ self-instrumentation to confirm data integrity, stream continuity, and temporal alignment; no human-subjects research is reported, and no behavioural, emotional, or accessibility outcomes are inferred. By focusing on infrastructure rather than inference, the proposed framework provides a reusable scaffold for future ethics-approved DHM studies, supporting inclusive and accessible research design through platform-independent sensing and interaction modelling. This paper is organised into the following sections: Related Works : reviews prior research in Digital Human Modelling, multimodal physiological sensing, game-based interaction, accessibility, and ethical considerations, positioning the present work within existing DHM and HCI literature while identifying limitations in portability, abstraction, and ethical separation. Framework Overview : introduces the design objectives and architectural principles of the proposed platform-agnostic DHM framework, including separation of sensing, interaction modelling, and inference readiness, with emphasis on accessibility-oriented and ethically bounded research design. Sensing Integration and Verification : describes the integration of the OpenBCI Galea headset as a multimodal sensing layer, detailing signal abstraction, temporal synchronisation, technical verification via author self-instrumentation, and considerations for scalability and data throughput. Interaction Modelling and Applied Implications : presents the game-based interaction environment and interaction primitives, followed by illustrative DHM and HCI use cases and concrete accessibility adaptation examples that demonstrate how the framework may support inclusive research without embedding evaluative or diagnostic assumptions. Conclusion : summarises the contribution and limitations of the framework and outlines planned ethics-approved validation steps and future research directions. 2 Related Works Human Modelling has a long history within ergonomics, safety, and human-system interaction, where computational representations of human characteristics are used to inform system design rather than to evaluate individual performance [ 4 , 5 ] . Early DHM research established that modelling need not be limited to visual or biomechanical avatars, but can instead operate at the level of interaction structure and task abstraction [ 6 ] . Layered DHM architectures separating data acquisition, abstraction, and modelling have subsequently been advocated to support reuse across application domains and experimental contexts [ 6 ] . In parallel, research in physiological computing has demonstrated that signals such as Electroencephalography (EEG), Electromyography (EMG), Electro-oculography (EOG), and cardiovascular measures can be incorporated into interactive systems as additional information channels [ 7 ] . Importantly, foundational work in this area treats physiological signals as interaction-level observables rather than direct indicators of internal cognitive or emotional state. Multimodal sensing approaches are commonly adopted to improve robustness and contextual coverage in wearable and human-centred systems [ 2 ] , although much of the literature focuses on downstream classification or inference, raising methodological and ethical considerations. Recent advances in wearable biosensing have enabled compact platforms that integrate multiple physiological and inertial modalities into a single device. The Galea headset [ 15 ] , for example, provides concurrent EEG, EMG, EOG, photoplethysmography (PPG), and inertial measurement streams intended for research and interactive applications [ 3 , 9 ] . Existing work using similar sensing technologies typically embeds these signals within task-specific pipelines, limiting portability and reuse across studies. Games and interactive simulations have also been widely used as structured environments [ 10 ] for studying human interaction. Digital games offer deterministic mechanics, repeatable task structures, and well-defined event boundaries, making them suitable as controlled interaction substrates [ 17 ] . Prior work has combined gameplay with physiological sensing to model affective or experiential states, often focusing on real-time interpretation or performance evaluation [ 11 , 12 ] . In contrast, more neutral uses of games treat them as task environments that generate structured interaction events without embedding interpretative assumptions, supporting reproducible modelling approaches. Accessibility and inclusion have increasingly been framed within HCI as systems-level design challenges rather than properties to be assessed post hoc [ 16 ] . Inclusive

Read the complete text