Towards Deeper Understanding of Natural User Interactions in Virtual Reality Based Assembly Tasks

Ryan Ghamandi University of Central Florida, Yahya Hmaiti University of Central Florida, Mykola Maslych University of Central Florida, Ravi Kiran Kattoju University of Central Florida, Joseph J. LaViola Jr. University of Central Florida · 2025

Read Towards Deeper Understanding of Natural User Interactions in Virtual Reality Based Assembly Tasks by Ryan Ghamandi University of Central Florida, Yahya Hmaiti University of Central Florida, Mykola Maslych University of Central Florida, Ravi Kiran Kattoju University of Central Florida, Joseph J. LaViola Jr. University of Central Florida (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17124.txt.

Introduction

Ryan Ghamandi and four colleagues study how people naturally talk and gesture to a robot during assembly tasks in virtual reality, in a 2025 paper. Participants worked with a simulated robot arm on LEGO building and circuit board tasks while a hidden experimenter controlled the robot's responses, a setup called Wizard-of-Oz. The team recorded voice, hand tracking, and gaze data throughout.

Instructive and collaborative tasks drew out different behavior, and strategies shifted as tasks went on. When a location was ambiguous, people used phrases like put that there; when it was clear, they gave fuller descriptions. The authors offer the patterns, plus their data set, as guidance for designing natural multimodal interfaces.

Catalog record

CREATORRyan Ghamandi University of Central Florida, Yahya Hmaiti University of Central Florida, Mykola Maslych University of Central Florida, Ravi Kiran Kattoju University of Central Florida, Joseph J. LaViola Jr. University of Central Florida
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-17124

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Deeper Understanding of Natural User Interactions in Virtual Reality Based Assembly Tasks Ryan Ghamandi University of Central Florida, Yahya Hmaiti University of Central Florida, Mykola Maslych University of Central Florida, Ravi Kiran Kattoju University of Central Florida, Joseph J. LaViola Jr. University of Central Florida Abstract We explore natural user interactions using a virtual reality simulation of a robot arm for assembly tasks. Using a Wizard-of-Oz study, participants completed collaborative LEGO and instructive PCB assembly tasks, with the robot responding under experimenter control. We collected voice, hand tracking, and gaze data from users. Statistical analyses revealed that instructive and collaborative scenarios elicit distinct behaviors and adopted strategies, particularly as tasks progress. Users tended to use ”put-that-there” language in spatially ambiguous contexts and more descriptive instructions in spatially clear ones. Our contributions include the identification of natural interaction strategies through analyses of collected data, as well as the supporting dataset, to guide the understanding and design of natural multimodal user interfaces for instructive interaction with systems in virtual reality. \onlineid TODO \vgtccategory Research \vgtcinsertpkg \teaser Task setups for LEGO and PCB tasks during completion. Introduction keywords Collaboration, virtual reality, augmented reality, natural user interfaces, 3D user interfaces, multimodal interaction Interaction with systems allows individuals to tell a system what to do or how to perform [ 67 ] . The medium through which users communicate with a system is the user interface, to which users provide inputs. Natural User Interfaces (NUIs) [ 53 ] take natural actions as input to a system, with Natural Multimodal User Interfaces (NMUIs) accepting multiple inputs at a time such as voice and gestures. These serve to significantly improve user experiences by supporting more intuitive, engaging, and effective ways to interact [ 39 , 50 ] . Work has gone towards creating NMUIs for industrial applications such as modeling and assembly, starting with early works that employ speech-based NUIs [ 61 , 7 ] . From there, they have evolved into NMUIs for scenarios such as virtual reality (VR) [ 23 , 72 , 32 ] , augmented reality [ 64 , 14 , 69 ] , and CAD modeling using gestural inputs for manual manipulation [ 21 , 22 ] . These interfaces enable users to instruct a system in the same way they directly interact with the environment. However, achieving this natural interaction poses challenges for delivering instructions effectively. Systems tend to require users to provide natural inputs in specific patterns. Without the necessary training and/or domain knowledge, this can lead to misunderstandings between the system and user due to misaligned inputs [ 29 ] . In the assembly domain, these interfaces support applications such as rapid prototyping, training, and robotic surgery, enabling seamless operation through natural user instruction [ 44 , 1 , 55 , 31 ] . Recent work in this domain has increased understanding of how users provide inputs to systems, but still have key limitations. Interactions are fueled by prior instruction provided to the user on how to interact with the system, leading to outcomes that represent taught behaviors rather than natural, intuitive interactions. Additionally, current research does not account for intuitive multimodal interactions, particularly in VR, where interaction modalities may vary from real-world tasks despite similar expected outcomes [ 37 , 57 ] . A lack of investigation into capturing sequences of raw, unstructured, and intuitive multimodal input highlights the need for a more nuanced understanding of user behavior in relation to the development of robust and adaptive NMUIs. To address this gap, we investigate how users naturally instruct a system to perform desired actions, we conducted a single-factor elicitation study to explore how users naturally interact with a virtual robot arm in VR assembly tasks. Participants completed two types of ecologically valid tasks with a Wizard-of-Oz controlled virtual robot arm. In the first task, collaborative LEGO assembly , both the participant and the robot manipulated LEGO to recreate a reference model, where LEGO acted as relative spatial anchors that made referencing easy. In the second task, an instructive Printed Circuit Board (PCB) assembly , participants instructed the robot to place PCB components on a blank board based on a reference design where the locations for placing objects would be spatially ambiguous to describe. VR was used as a medium to aid in data collection. Our analysis revealed that users predominantly adopted Bolt’s ”put-that-there” method [ 7 ] , with different language structures of the same commands being delivered. Instruction delivery also varied sequentially, aligning with findings from Oviatt et al. [ 50 , 47 ] . In addition to these findings, we found that descriptive commands and spatially vague instructions accompanied by spatial cues were common in the LEGO and PCB task respectively. Also, participants used more unimodal commands during the collaborative task, more trimodal commands during the instructive task, and bimodal commands in both. Finally, the method by which users would reference locations in the environment for object placement had both spatial and temporal dependencies which was rooted in the availability of concrete spatial anchors. We list our main contributions as follows: (1) Key observations of tendencies and statistical analyses of user interactions with a simulated robot arm during VR assembly tasks . (2) Practical design recommendations to enhance multimodal interactions with NMUIs in assembly tasks . (3) A dataset with annotated human interactions involving a simulated robot arm during VR assembly tasks which provides insights into how users interacted in both tasks . 1 Related Work In this section, we detail previous works relevant to our study: multimodal input systems and elicitation studies. We review recent works in these fields and identify gaps that our work aims to address. 1.1 Multimodal Input Systems and Interfaces Multimodal Input Systems and Interfaces have been developed and used for decades, with fundamentals focused on how modalities interact with each other in more traditional interfaces like mouse and keyboard [ 49 , 51 , 48 ] . The paradigms of early, late, and hybrid fusion are detailed in these works in terms of how inputs are fused. More advanced interfaces include ones that integrate speech and gestures [ 16 , 20 , 36 , 50 ] . These works use classification methods such as neural networks and typed feature structures to accept inputs in specific structures while allowing some degree of variability of input sequence. Findings from the use of these systems are based on spatio-temporal patterns such as inputs like speech and gesture being sequential rather than simultaneous, as well as statements being grounded in context by other relevant modalities. Recent natural interfaces have been designed for applications such as robot interaction [ 65 ] and VR [ 43 ] , with emphasis also placed on task collaboration [ 24 , 25 ] . This includes more recent work that has also been done to develop natural input models capable of learning the intricacies of these inputs. Our work extends prior research and differs from it by creating an environment to capture a wide array of sequences of multiple modalities, where users are untrained and complex multimodal instructions are accepted. Additionally, the sequences of modalities captured are context-aware and adaptive in order to account for previous interactions and

Read the complete text