ClickAIXR: On-Device Multimodal Vision-Language Interaction with Real-World Objects in Extended Reality PubID: pubid: 0000–0000/00$00.00 © 2021 IEEE
Dawar Khan, Alexandre Kouyoumdjian, Xinyu Liu, Omar Mena, Dominik Engel, Ivan Viola Figure 1 : Overview of ClickAIXR . Left: A user selects a real-world object and queries the on-device VLM (e.g., “What is this?”). Middle: The object selection interface, where users adjust a red cropping box via three sliders controlling depth, width, and height. Right: Examples of selected objects used in our experiments. · 2026
Read ClickAIXR: On-Device Multimodal Vision-Language Interaction with Real-World Objects in Extended Reality PubID: pubid: 0000–0000/00$00.00 © 2021 IEEE by Dawar Khan, Alexandre Kouyoumdjian, Xinyu Liu, Omar Mena, Dominik Engel, Ivan Viola Figure 1 : Overview of ClickAIXR . Left: A user selects a real-world object and queries the on-device VLM (e.g., “What is this?”). Middle: The object selection interface, where users adjust a red cropping box via three sliders controlling depth, width, and height. Right: Examples of selected objects used in our experiments. (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-04905.txt.
Introduction
Khan and five colleagues build ClickAIXR, a system that lets people in extended reality click a real object and ask about it in plain language. Unlike cloud-based tools or gaze-only selection, it runs a vision-language model on the device and uses controller pointing for precise choice. Once clicked, the object image is processed locally, and answers arrive as text and speech. Local inference cuts ambiguity, protects privacy, and lowers delay, and a user study compares the tool with gaze-based rivals.
Catalog record
| CREATOR | Dawar Khan, Alexandre Kouyoumdjian, Xinyu Liu, Omar Mena, Dominik Engel, Ivan Viola Figure 1 : Overview of ClickAIXR . Left: A user selects a real-world object and queries the on-device VLM (e.g., “What is this?”). Middle: The object selection interface, where users adjust a red cropping box via three sliders controlling depth, width, and height. Right: Examples of selected objects used in our experiments. |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41144 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.04905
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ClickAIXR: On-Device Multimodal Vision-Language Interaction with Real-World Objects in Extended Reality PubID: pubid: 0000–0000/00$00.00 © 2021 IEEE Dawar Khan, Alexandre Kouyoumdjian, Xinyu Liu, Omar Mena, Dominik Engel, Ivan Viola Figure 1 : Overview of ClickAIXR . Left: A user selects a real-world object and queries the on-device VLM (e.g., “What is this?”). Middle: The object selection interface, where users adjust a red cropping box via three sliders controlling depth, width, and height. Right: Examples of selected objects used in our experiments. Abstract We present ClickAIXR, a novel on-device framework for multimodal vision-language interaction with objects in extended reality (XR). Unlike prior systems that rely on cloud-based AI (e.g., ChatGPT) or gaze-based selection (e.g., GazePointAR), ClickAIXR integrates an on-device vision-language model (VLM) with a controller-based object selection paradigm, enabling users to precisely click on real-world objects in XR. Once selected, the object image is processed locally by the VLM to answer natural language questions through both text and speech. This object-centered interaction reduces ambiguity inherent in gaze- or voice-only interfaces and improves transparency by performing all inference on-device, addressing concerns around privacy and latency. We implemented ClickAIXR in the Magic Leap SDK (C API) with ONNX-based local VLM inference. We conducted a user study comparing ClickAIXR with Gemini 2.5 Flash and ChatGPT 5, evaluating usability, trust, and user satisfaction. Results show that latency is moderate and user experience is acceptable. Our findings demonstrate the potential of click-based object selection combined with on-device AI to advance trustworthy, privacy-preserving XR interactions. The source code and supplementary materials are available at nanovis.org/ClickAIXR.html . I Introduction Extended reality (XR) environments present unprecedented opportunities for seamless integration of digital information with the physical world. As users navigate real-world scenarios – whether exploring historical sites, learning complex concepts, or requiring assistance with visual accessibility – the ability to instantly query and understand their visual environment becomes increasingly valuable. Vision Language Models (VLMs) offer compelling capabilities to bridge this gap by enabling natural language interactions with visual content, transforming how users can gain knowledge about what is in their field of view. Current approaches to integrating VLMs into XR applications rely on cloud-based inference, exemplified by systems like GazePointAR [ 16 ] , and XaiR [ 28 ] . While these cloud-based solutions leverage powerful computational resources to deliver high-quality responses, they also introduce significant challenges that limit their practical deployment and widespread adoption. Most critically, transmitting potentially sensitive visual data from users’ personal environments to proprietary cloud services raises substantial privacy concerns, particularly in educational, workplace, or personal contexts where confidential information is abundant. Beyond privacy considerations, cloud-based deployments suffer from inherent latency, increased power consumption, recurring subscription costs, and dependence on a stable network connection. Another challenge lies in how to feed the VLM with target images or a region of interest (ROI) . In addition, voice-based interaction is prone to pronoun ambiguity [ 29 ] and lacks context awareness. For example, when a user provides an object to the VLM and asks “What is that?”, the pronoun that introduces confusion, as the model cannot confirm which object the user is referring to. This issue has recently been addressed by GazePointAR [ 16 ] , which leverages eye gaze, finger pointing, and a cloud-based VLM to enable ambiguity-free voice assistance and XR interaction. However, two key challenges remain: (i) GazePointAR is cloud-based, so all of the aforementioned limitations persist, and (ii) it relies on YOLOv8 [ 13 ] for object segmentation, which introduces additional processing time (reported CV stage: 3.75\pm 0.23 s) [ 16 ] . This paper presents a novel approach, ClickAIXR , that addresses these limitations through local on-device VLM deployment on XR headsets (Magic Leap 2), leveraging the increased computational power of recent advances in wearable hardware. Our method reduces long, uninterrupted waiting times by bypassing time-consuming image segmentation preprocessing. Instead, it provides users with interactive selection of regions of interest through our Gaze-Locked Clipping Window (GCW). The GCW follows user gaze to position a clipping window on the target object and offers three slider-based GUI controls to adjust the width, height, and depth (distance from the user) of the window (see , middle), enabling accurate object selection. After selecting a ROI , users can pose natural language questions about the chosen content. By performing inference entirely on-device, our system eliminates the need to transmit sensitive visual data to proprietary AI providers, thereby ensuring complete privacy preservation. Despite operating on constrained hardware compared to cloud-based solutions, our approach achieves comparable overall latency with systems like GazePointAR, by removing network delays and pre-processing overhead. Deploying the VLM locally also offers significant advantages in power efficiency, contributing to more sustainable AI applications. We evaluate our method in a within-subjects user study with 12 participants, where we compare ClickAIXR to Google Gemini and OpenAI ChatGPT, and we show that on-device VLM inference can deliver practical and privacy-preserving XR applications with tolerable compromises on user experience. To summarize, our work makes the following contributions: • We present (to the best of our knowledge) the first on-device multimodal VLM application for XR that supports voice, eye–gaze, text, and image inputs. Running entirely on-device preserves privacy and enables application-specific fine-tuning without sharing data with third parties. The framework is model-agnostic and supports deploying any suitably sized VLM in ONNX format with only minor changes to the inference and tokenizer modules. • We introduce Gaze-Locked Clipping Window (GCW), a gaze-locked, controller-adjustable rectangular window that precisely selects a target object in XR (see Fig. , middle). • We conduct a user study and a timing analysis comparing against cloud-based baselines (i.e., ChatGPT, Gemini). Our system achieves acceptable usability and practical performance, with mean inference time of 5.36 s (Books)–5.48 s (COCO) per image and a consistent token-generation speed of 3.36 tokens/s . The remainder of the paper first explores the related work on deploying VLMs on-device and multimodal interaction in XR, followed by our method where we present our solution to the on-device deployment and how we solve for pronoun ambiguity. After that we present and discuss the user study we performed to evaluate our approach and present our conclusions. II Related Work We review related work along two key research directions: (1) the integration of vision-language models (VLMs) for on-device inference, and (2) the design of human interaction techniques that enable natural, multimodal communication with objects in extended reality (XR). The first line of work emphasizes the importance of privacy, latency, and responsiveness in VLM-powered systems, while the second focuses on resolving referential ambiguity and improving interaction flow through modalities such as gaze, speech, and visual input. II-A On-Device Vision Language Models Recent advances in VLMs have significantly improved the ability of systems to understand and generate language grounded in visual context. Models such as BLIP-2 [ 20 ] and LLaVa [ 22 ] demonstrate strong visual reasoning. While such models provide robust multimodal capabilities, they are typically designed for server-grade inference. MobileVLM [ 4 ] introduces a family of compact, instruction-following VLMs tailored for mobile CPUs and edge GPUs. Similarly, MiniGPT-4 [ 33 ] and InstructBLIP [ 5 ] aim to reduce complexity and improve instruction-following in visual contexts, but still largely depend on training resources and time. Recent