Increasing Interaction Fidelity: Training Routines for Biomechanical Models in HCI Conference: The 38th Annual ACM Symposium on User Interface Software and Technology; September 28-October 1, 2025; Busan, Republic of Korea The 38th Annual ACM Symposium on User Interface Software and Technology (UIST

Michał Patryk Miazga, Patrick Ebel · 2025

Read Increasing Interaction Fidelity: Training Routines for Biomechanical Models in HCI Conference: The 38th Annual ACM Symposium on User Interface Software and Technology; September 28-October 1, 2025; Busan, Republic of Korea The 38th Annual ACM Symposium on User Interface Software and Technology (UIST by Michał Patryk Miazga, Patrick Ebel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-16581.txt.

Introduction

Michal Miazga and Patrick Ebel improve training routines for biomechanical models in a 2025 human computer interaction paper. Forward simulation of the body could generate human like movement for interactive tasks. Training such models with reinforcement learning is hard, especially for precise finger movements on touchscreens.

Current methods limit fidelity, simplify the body model, and generalize poorly. Using a touchscreen pointing task, the authors show that curriculum learning, action masking, richer network settings, and small simulation tweaks cut training time and raise fidelity. Their routines allow more complex body models and offer practical tips for researchers.

Catalog record

CREATORMichał Patryk Miazga, Patrick Ebel
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE12 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-16581

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Increasing Interaction Fidelity: Training Routines for Biomechanical Models in HCI Conference: The 38th Annual ACM Symposium on User Interface Software and Technology; September 28-October 1, 2025; Busan, Republic of Korea The 38th Annual ACM Symposium on User Interface Software and Technology (UIST Adjunct ’25), September 28-October 1, 2025, Busan, Republic of Korea DOI: 10.1145/3746058.3758385 ISBN: 979-8-4007-2036-9/2025/09 CCS: Human-centered computing User models CCS: Human-centered computing Pointing CCS: Human-centered computing Systems and tools for interaction design CCS: Computing methodologies Reinforcement learning Michał Patryk Miazga, Patrick Ebel Abstract . Biomechanical forward simulation holds great potential for HCI, enabling the generation of human-like movements in interactive tasks. However, training biomechanical models with reinforcement learning is challenging, particularly for precise and dexterous movements like those required for touchscreen interactions on mobile devices. Current approaches are limited in their interaction fidelity, require restricting the underlying biomechanical model to reduce complexity, and do not generalize well. In this work, we propose practical improvements to training routines that reduce training time, increase interaction fidelity beyond existing methods, and enable the use of more complex biomechanical models. Using a touchscreen pointing task, we demonstrate that curriculum learning, action masking, more complex network configurations, and simple adjustments to the simulation environment can significantly improve the agent’s ability to learn accurate touch behavior. Our work provides HCI researchers with practical tips and training routines for developing better biomechanical models of human-like interaction fidelity. Figure 1 . Overview of training routines to improve RL-based biomechanical forward simulations. 1. Introduction Table 1. Evaluation results represent the mean values of 100 episodes, each with a maximum duration of 10 seconds. Network Size Max Timesteps Action Masking Curriculum Learning Stage 2 Dynamic Reward Stage 2 Early Reward Button Radius (mm) Success Rate Avg. Errors / Success Episode Avg. Time / Success Episode (s) 128 \times 128 200M ✓ ✓ ✗ ✓ 1.5 82% 176.08 3.39 256 \times 256 200M ✓ ✓ ✓ ✗ 1.5 99% 156.81 3.35 256 \times 256 800M ✗ ✓ ✓ ✗ 1.5 5% 3.6 1.6 512 \times 512 200M ✓ ✓ ✓ ✗ 1.5 100% 8.07 1.40 512 \times 512 200M ✓ ✗ ✗ ✗ 6 0% 0 ✗ 512 \times 512 200M ✓ ✗ ✗ ✓ 1.5 64% 179.56 3.62 Simulated users that model human-like, dexterous interactions with technology hold great potential to make user-centered design more efficient, rigorous, and predictable ( Murray-Smith et al. 2022 ; Lorenz et al. 2024 ) . At the core of these systems are biomechanical models that capture anatomical and physiological detail. Advances in computational power have made it possible to use increasingly complex models, such as those in MyoSuite ( Caggiano et al. 2022 ; Wang et al. 2022 ) and User-in-the-Box ( Ikkala et al. 2022 ; Ikkala and Hämäläinen 2020 ) , which can simulate muscle fatigue or neuromuscular diseases. However, training Reinforcement Learning (RL) agents to learn muscle-level control in physically simulated environments remains a highly complex task. Training is often brittle, slow, and difficult to reproduce. This limits interaction fidelity and accordingly the broader adoption of biomechanical modeling in the Human-Computer Interaction (HCI) community for technology design and evaluation. For instance, in User-in-the-Box ( Ikkala et al. 2022 ; Fischer et al. 2021 ) , the pointing targets had a radius of 5–15 cm. Given that the iPhone 16 has a diagonal of 15.5 cm this is large, especially when such models should be applied to mobile use cases. Furthermore, the action space for arm movements was simplified by disabling wrist flexion and various muscles, effectively reducing the model to five degrees of freedom and 26 active muscles. In Sim2VR , Fischer et al. 2024 also reduced the number of muscles and replaced some with joint torque actuators, which are easier to control. Their model also focuses primarily on coarse movements in VR. Overall, current approaches demonstrate the potential of musculoskeletal forward simulation for HCI , but they fall short of enabling the precise control required for most interactive tasks. We argue that these shortcomings are primarily due to ineffective training routines. Although reward shaping has been studied ( Selder et al. 2025 ; Nowakowski et al. 2021 ; Chen et al. 2023 ) , the influence of other training routines on the learning of fine motor skills remains underexplored. In this work, we present and evaluate performance improvements achieved through action masking, curriculum learning, alternative policy network configurations, and task-specific improvements to the simulation environment. Based on our findings, we propose actionable training routines that lower technical barriers and support the development of high-fidelity, human-like control in physics-based biomechanical simulations for HCI . 2. Training Routines for Biomechanical Models Our primary goal is to learn stable muscle-level control policies that enable precise and dexterous movements. All evaluations are conducted using Proximal Policy Optimization (PPO) ( Schulman et al. 2017 ) , and their performance is assessed based on the success rate of touch interactions. We consider an interaction as successful if the target element is touched within 10 seconds after the start of the episode. We further count unintended interactions outside the target area (e.g., due to overshooting) as errors, indicating suboptimal control and a lack of fine motor skills. In the following, we introduce all the routines evaluated in this work. Action Masking As shown in prior work ( Huang and Ontañón 2020 ; Miazga et al. 2025 ; Stolz et al. 2024 ; Fischer et al. 2024 ; Ikkala et al. 2022 ) , action masking is an effective technique to reduce the dimensionality of the action space, limit unnecessary exploration, and accelerate learning. In our implementation, we apply action masking by disabling all fingers except the index finger. Our results (see Table 1 ) show that action masking highly improves training efficiency. Multi-Stage Curriculum Learning In RL , curriculum learning improves learning efficiency and final performance by training agents on tasks of gradually increasing difficulty ( Narvekar et al. 2020 ; Narvekar 2017 ; Luo et al. 2020 ) . To achieve the fine motor control required for touchscreen interactions, we introduce a structured, multi-stage curriculum. Each curriculum stage increases complexity via sub-stages. The agent advances to the next (harder) stage only once it meets a performance threshold in the final sub-stage of the current curriculum stage. As shown in Table 1 and illustrated in Figure 1 , the following curriculum stages significantly improve training efficiency: Stage 1: Increasing Task Complexity. To simplify early training, the agent begins from a fixed position and must reach a large, button-shaped 3D target placed just above the surface with its fingertip. As training progresses, the 3D target gradually flattens into a 2D surface (see Figure 1 ), shifting the task from interacting with a volumetric object to a more demanding surface-based interaction that requires greater precision. Stage 2: Dynamic Reward-Shaping. Reward shaping involves modifying the reward function to guide the agent toward desired behaviors more effectively ( Hu et al. 2020 ; Grzes 2017 ; Vamplew et al. 2022 ; Dewey 2014 ; Eschmann 2021 ) . Our experiments show that starting with a simple reward to then gradually increase reward complexity, based on the training progress, significantly improves performance. Early rewards penalize incorrect button presses, while later

Read the complete text