HandOver: Enabling Precise Selection & Manipulation of 3D Objects with Mouse and Hand Tracking Conference: The 38th Annual ACM Symposium on User Interface Software and Technology; September 28-October 1, 2025; Busan, Republic of Korea The 38th Annual ACM Symposium on User Interface Software and Tech
Esen K. Tütüncü, Mar Gonzalez-Franco, Eric J. Gonzalez · 2025
Read HandOver: Enabling Precise Selection & Manipulation of 3D Objects with Mouse and Hand Tracking Conference: The 38th Annual ACM Symposium on User Interface Software and Technology; September 28-October 1, 2025; Busan, Republic of Korea The 38th Annual ACM Symposium on User Interface Software and Tech by Esen K. Tütüncü, Mar Gonzalez-Franco, Eric J. Gonzalez (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00211.txt.
Introduction
Esen Tütüncü, Mar Gonzalez-Franco, and Eric Gonzalez blend mouse and hand tracking for virtual reality work, in a 2025 paper. Their system, HandOver, targets a common trade-off: mice aim precisely but cannot grab, while hand tracking grabs naturally but aims loosely.
With HandOver, the mouse drives a depth-aware cursor for exact targeting. Resting a hand over the mouse then switches seamlessly to direct hand control of the picked object. In a formal study, volunteers docked objects in three dimensions using HandOver and two ray-based rivals. HandOver produced fewer errors at every distance. Posture analysis and self-reports also favored it for comfort and workload. The authors argue that mixing old and new input styles beats forcing one style to do everything. Immersive tools, they suggest, should let each device do what it does best.
Catalog record
| CREATOR | Esen K. Tütüncü, Mar Gonzalez-Franco, Eric J. Gonzalez |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-00211
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HandOver: Enabling Precise Selection & Manipulation of 3D Objects with Mouse and Hand Tracking Conference: The 38th Annual ACM Symposium on User Interface Software and Technology; September 28-October 1, 2025; Busan, Republic of Korea The 38th Annual ACM Symposium on User Interface Software and Technology (UIST ’25), September 28-October 1, 2025, Busan, Republic of Korea DOI: 10.1145/3746059.3747757 ISBN: 979-8-4007-2037-6/2025/09 CCS: Human-centered computing Gestural input CCS: Human-centered computing Virtual reality Esen K. Tütüncü, Mar Gonzalez-Franco, Eric J. Gonzalez Abstract . We present HandOver, an extended reality (XR) interaction technique designed to unify the precision of traditional mouse input for object selection with the expressiveness of hand-tracking for object manipulation. With HandOver, the mouse is used to drive a depth-aware 3D cursor enabling precise and restful targeting — by hovering their hand over the mouse, the user can then seamlessly transition into direct 3D manipulation of the target object. In a formal user study, we compare HandOver against two ray-based techniques: traditional raycasting (Ray) and a hybrid method (Ray+Hand) in a 3D docking task. Results show HandOver yields lower task errors across all distances, and moreover improves interaction ergonomics as highlighted by a RULA posture analysis and self-reported measures (NASA-TLX). These findings illustrate the benefits of blending traditional precise input devices with the expressive gestural inputs afforded by hand-tracking in XR, leading to improved user comfort and task performance. This blended paradigm yields a unified workflow allowing users to leverage the best of each input modality as they interact in immersive environments. Figure 1 . The HandOver technique enables a smooth transition from mouse-based object selection to direct hand manipulation. Interaction stages: (Left) Selecting an object with the mouse, (Middle) Entering the handOver state by lifting the hand over the mouse, and (Right) Grasping the object with the hand. Three different states of the Distant Hand Interaction † † cc-license: by 1. Introduction Interaction research has long faced the challenge of combining the precision of traditional devices with the naturalness of spatial interaction. This challenge remains largely unsolved, creating an opportunity for novel approaches like the one we present in this paper. Classic mouse-based input remains remarkably powerful for planar tasks, thanks to decades of refinements in input technologies and techniques ( Hinckley 2007 ) , as well as foundational insights into human motor performance ( Fitts 1954 ) . However, integrating mouse input into 3D environments presents fundamental challenges: primarily the difficulty of mapping 2D planar input to 3D spatial interactions, creating depth ambiguity, and managing proper visual feedback in immersive contexts ( Fortune 2017 ) . Previous approaches have attempted to solve these problems through various techniques, but no comprehensive solution has emerged ( Nguyen et al. 2023 ; Bergström et al. 2021 ) . 3D interaction can be better understood when divided into three distinct phases: selection, confirmation, and manipulation. Each phase presents unique challenges in spatial environments. The act of selecting and confirming an object often benefits from precise pointing or clicking—areas where mouse input excels. Yet once an object is “in hand,” direct control over its position and rotation is more natural with gesture-based 3D input or ray-casting techniques ( Poupyrev et al. 1996 ; Pfeuffer 2024 ) . Even so, purely gestural approaches can cause tracking jitter, discomfort over long sessions, or confusion when users must isolate a small target amidst clutter ( Tatzgern et al. 2016 ) . This tension suggests that no single approach has fully bridged the needs of precise selection, smooth confirmation, and fluid manipulation at varying distances. Meanwhile, many authors have stressed that 3D interfaces have unique constraints not present in 2D. Reviews of target selection in virtual reality ( Huang et al. 2019 ) highlight this tension, pointing out that robust user performance typically demands a stable anchor, such as a mouse, for small precise targets, and more direct arm gestures for large or close ones. Some propose bridging these differences through fidelity models ( Bonfert et al. 2024 ) or “office of the future” approaches ( Grubert et al. 2019 ) that seamlessly integrate standard desktop gear into XR. Others argue that carefully balancing near-field “mouse precision” with far-field “arm extension” is crucial ( Bowman and Hodges 1997 ) , as purely gestural solutions often amplify small tremors over distance ( Mayer et al. 2018 ) or fail to provide the tactile anchoring a conventional mouse inherently offers. To address this, we propose HandOver , a hybrid approach that integrates precise mouse-based selection with natural hand-based manipulation. HandOver combines a depth-aware cursor for far-field targeting with a virtual "hand clone" that projects natural gestures to distant objects. By leveraging mouse-based input for fine-grained targeting and using hand tracking for more intuitive manipulation, HandOver seeks to unify the user’s sense of manual control from initial selection through final placement. The system dynamically transitions between these modalities so that small or closely spaced objects can be handled with mouse-level precision, while large or nearby objects can be manipulated via direct “hands-on” interaction. We conducted a user study to compare HandOver against two methods: Ray+Hand, a ray-based interaction technique that maintains consistent control-display gain regardless of distance, and Ray, the industry standard raycast interaction. We evaluated performance metrics such as targeting time, docking time, error, and physical workload across near, mid, and far distances, along with ergonomics. Our results shed light on how to optimize 3D interaction techniques by balancing familiarity (mouse-based selection) with the inherent manual expressiveness of gestural input, highlighting the hand’s foundational place in digital interfaces. 2. Related Work 2.1. Mouse-Based Precision in 3D Interfaces The mouse remains one of the most precise input devices in computing, and its core strength—fine-grained control—has led to ongoing efforts to bring its benefits into 3D environments. Researchers have experimented with embedding mouse-like control into spatial interfaces, exploring how workflows from 2D design tools might translate into immersive settings. For instance, architectural tools have incorporated switchable 3D cursors to enable collaborative co-creation ( Dorta et al. 2016 ) . Others have examined techniques for toggling between absolute and relative pointing, revealing how traditional input can remain effective in multi-dimensional tasks ( Forlines et al. 2006 ) . More recently, desktop mouse input has been reintroduced into VR, where it has shown advantages over mid-air gestures, particularly in tasks that demand delicate alignment ( Zhou et al. 2022 ) . 2.2. Precision-Enhancing Pointing Techniques Beyond mouse emulation, a number of approaches aim to preserve or improve pointing accuracy in spatial settings through dynamic adaptation. Modulating the control-display gain based on target distance or size has been shown to support more precise selection ( König et al. 2009 ) . In augmented reality, head- and eye-based methods have been evaluated for their ability to replicate the targeting fidelity of mouse input ( Kytö et al. 2018 ) . However, precision degrades rapidly when users move away from the body or toward large displays. Studies of distant freehand pointing suggest that while the mouse excels in near-field contexts, it often requires complementary methods to maintain accuracy at scale ( Vogel and Balakrishnan 2005 ) . 2.3. Ray-Based Selection and Its Limitations Ray-based interaction is one of the most common strategies for selecting distant objects in VR.