The Midas Touch in Gaze vs. Hand Pointing: Modality-Specific Failure Modes and Implications for XR Interfaces
Mohammad Dastgheib, Fatemeh Pourmahdian · 2026
Read The Midas Touch in Gaze vs. Hand Pointing: Modality-Specific Failure Modes and Implications for XR Interfaces by Mohammad Dastgheib, Fatemeh Pourmahdian (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15991.txt.
Introduction
Dastgheib and Pourmahdian compare gaze and hand pointing for XR interfaces. Hand input tires users, while gaze input selects what it merely touches. Their web platform runs a multidirectional tapping task. Gaze declutter and hand target widening serve as adaptive aids. Sixty-nine people crossed modality, mode, and pressure factors. Hands beat gaze on speed, errors, and workload scores. Gaze errors were nearly all slips, at 99.2 percent. Hand errors were mostly misses, at 95.7 percent. Only the gaze aid ran clean in this dataset.
Catalog record
| CREATOR | Mohammad Dastgheib, Fatemeh Pourmahdian |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52116 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.15991
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Midas Touch in Gaze vs. Hand Pointing: Modality-Specific Failure Modes and Implications for XR Interfaces Mohammad Dastgheib, Fatemeh Pourmahdian Abstract Extended Reality (XR) interfaces impose both ergonomic and cognitive demands, yet current systems often force a binary choice between hand-based input, which can produce fatigue, and gaze-based input, which is vulnerable to the Midas Touch problem and precision limitations. We introduce the xr-adaptive-modality-2025 platform, a web-based open-source framework for studying whether modality-specific adaptive interventions can improve XR-relevant pointing performance and reduce workload relative to static unimodal interaction. The platform combines physiologically informed gaze simulation, an ISO 9241-9 multidirectional tapping task, and two modality-specific adaptive interventions: gaze declutter and hand target-width inflation. We evaluated the system in a 2 × 2 × 2 within-subjects design manipulating Modality (Hand vs. Gaze), UI Mode (Static vs. Adaptive), and Pressure (Yes vs. No). Results from N=69 participants show that hand yielded higher throughput than gaze (5.17 vs. 4.73 bits/s), lower error (1.8% vs. 19.1%), and lower NASA-TLX workload. Crucially, error profiles differed sharply by modality: gaze errors were predominantly slips (99.2%), whereas hand errors were predominantly misses (95.7%), consistent with the Midas Touch account. Of the two adaptive interventions, only gaze declutter executed in this dataset; it modestly reduced timeouts but not slips. Hand width inflation was not evaluable due to a UI integration bug. These findings reveal modality-specific failure modes with direct implications for adaptive policy design, and establish the platform as a reproducible infrastructure for future studies. Keywords: Extended Reality, Adaptive Interfaces, Gaze Interaction, Fitts’s Law, Multimodal Interaction, Gaze Simulation, Midas Touch, Linear Ballistic Accumulator, Intent Disambiguation 1 Introduction XR interaction changes the control demands of pointing by moving input from desktop devices to embodied hand and gaze actions. Unlike the desktop metaphor, where interaction is mediated by low-effort devices like the mouse and keyboard, XR requires the user to engage their entire body; the primary pointing devices are the user’s own hands and eyes. This embodied interaction is fraught with ergonomic and cognitive challenges. The “Gorilla Arm” syndrome, a phenomenon where prolonged mid-air arm extension leads to rapid musculoskeletal fatigue and pain, remains a critical barrier to the long-term adoption of gestural interfaces. Conversely, gaze-based interaction, which leverages the speed of the human oculomotor system, suffers from the “Midas Touch” problem—the inherent ambiguity between looking for perception and looking for action—and lacks the fine motor precision required for granular manipulation tasks (Jacob, 1990). Current XR systems typically force a binary choice: the user must either commit to a controller-based paradigm, accepting the physical fatigue, or a gaze-based paradigm, accepting the lack of precision and the potential for inadvertent triggers. This rigid dichotomy ignores the dynamic nature of human attention and the varying demands of different tasks. A high-precision manipulation task may require the stability of a hand controller, while a rapid visual search task is best served by the saccadic speed of the eye. To investigate whether adaptive support can alleviate these tradeoffs, we introduce the xr-adaptive-modality-2025 platform, a research framework for studying modality-specific adaptive support within hand and gaze input in XR-relevant pointing tasks. The system combines an ISO 9241-9 pointing task, physiologically informed gaze simulation, and modality-specific adaptive interventions intended to address distraction during gaze selection and spatial difficulty during hand selection. The central question is whether modality-specific adaptive support can improve performance and workload relative to static unimodal interaction (Soukoreff & MacKenzie, 2004). This paper makes three contributions. First, we present an open-source platform for controlled, reproducible study of gaze and hand pointing in XR-relevant tasks, including a physiologically informed gaze simulation, a policy-driven adaptation engine, and a full remote data collection infrastructure. Second, we provide quantitative empirical evidence on gaze vs. hand input in an ISO 9241-9 pointing task (N=69), showing that hand substantially outperforms gaze on throughput and error rate, and that the dominant failure modes are modality-specific: slips (false activations) for gaze and misses (spatial targeting failures) for hand. Third, we demonstrate that a simple gaze-declutter adaptation modestly reduces timeouts but does not address the core slip-based failure mode of gaze interaction, which has direct implications for designing modality-specific adaptive interventions. Unlike prior gaze+hand paradigms that treat the two modalities as complementary components of a fixed multimodal combination (e.g., gaze for acquisition + hand for confirmation (Jacob, 1990)), this work investigates modality-specific adaptive support —when and how the system should activate adaptations (declutter, width inflation) within each modality based on real-time performance signals. The platform could support future studies of modality switching. The study is guided by three research questions. RQ1 (Performance): Does a context-aware adaptive system yield higher throughput ( TP ) than static unimodal systems? RQ2 (Workload): Can modality-specific adaptive interventions reduce “Physical Demand” and “Frustration” (NASA-TLX) compared to static conditions? RQ3 (Adaptation): Do adaptive interventions improve performance and reduce workload relative to static conditions? (In this dataset, only gaze declutter was evaluable; hand width inflation was not applied due to a UI integration bug.) 2 Background and Related Work 2.1 Sensorimotor Implications of Hand and Gaze Input To design an effective adaptive system, one must first deconstruct the physiological mechanisms of the component modalities. Hand and gaze afford different control properties in XR: hand input supports precise corrective control, whereas gaze supports rapid orienting but introduces ambiguity when fixation is used for selection. Manual Pointing in XR: Manual input in XR, whether through held controllers or optical hand tracking, mimics the act of physical pointing. This interaction style benefits from proprioception—the body’s innate sense of limb position—which allows for high-precision corrections without visual attention. However, the biomechanical cost is substantial. In a 1:1 mapped XR environment, reaching a virtual object requires a corresponding physical motion. Frequent large-amplitude movements lead to fatigue in the deltoids and trapezius muscles. As fatigue sets in, the signal-to-noise ratio of the motor system degrades; the hand begins to tremor, increasing the effective target width required for accurate selection and reducing the overall throughput of the interaction. Gaze Interaction: The eye is the fastest motor organ in the human body. Saccades—rapid, ballistic movements of the eye—can reach velocities exceeding 900 degrees per second (Bahill & Stark, 1979), making gaze an incredibly efficient modality for target acquisition. However, the eye is fundamentally an input organ, not an output device. Using gaze for selection introduces several critical issues: (1) The Midas Touch —in the physical world, we can look at an object without interacting with it, but in a gaze-controlled interface, looking becomes equivalent to touching, requiring “dwell time” mechanisms that slow interaction (Jacob, 1990); (2) Microsaccades and Jitter —even when “fixated,” the eye performs microsaccades to refresh the retinal image (Martinez-Conde et al., 2004), meaning a gaze cursor is inherently noisy, making selection of small targets frustrating without smoothing algorithms; (3) Saccadic Suppression —during rapid eye movements, the visual system suppresses input to prevent motion blur (Bridgeman et al., 1975), creating a “blind” phase that makes the initial phase of gaze targeting effectively open-loop. 2.2 Signal Processing and Cognitive Load Framing To study these dynamics with experimental rigor and reproducibility, our platform employs a psychophysically-grounded generative simulation of gaze behavior rather than raw sensor input or hardware eye-tracking. Unlike raw eye-tracker signals—which conflate device-specific artifacts with genuine oculomotor