Can Foundation Models Revolutionize Mobile AR Sparse Sensing? DOI: XXXXXXX.XXXXXXX Conference: The 27th International Workshop on Mobile Computing Systems and Applications; February 25–26, 2026; Atlanta, GA ISBN: 978-1-4503-XXXX-X/2018/06 97 CCS: Computing methodologies Mixed / augmented reality CCS
Yiqin Zhao, Tian Guo · 2025
Read Can Foundation Models Revolutionize Mobile AR Sparse Sensing? DOI: XXXXXXX.XXXXXXX Conference: The 27th International Workshop on Mobile Computing Systems and Applications; February 25–26, 2026; Atlanta, GA ISBN: 978-1-4503-XXXX-X/2018/06 97 CCS: Computing methodologies Mixed / augmented reality CCS by Yiqin Zhao, Tian Guo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-02215.txt.
Introduction
Yiqin Zhao and Tian Guo ask whether the large pretrained foundation models reshaping computing can also rescue mobile sensing from an old trade-off. Phones and headsets cannot record and process everything their sensors see; power and computation are limited. Sparse sensing records only a subset of the data, which saves resources but injects uncertainty wherever information is missing across space and time.
Using real mobile augmented reality data, the authors evaluated foundation models on geometry-aware image warping, the technique that reuses information between camera frames. The models significantly improved warping quality and also led in three-dimensional scene reconstruction tests, and the approach scaled well. The paper closes by mapping open challenges that stand between these results and dependable sparse sensing systems on everyday devices.
Catalog record
| CREATOR | Yiqin Zhao, Tian Guo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-02215
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can Foundation Models Revolutionize Mobile AR Sparse Sensing? DOI: XXXXXXX.XXXXXXX Conference: The 27th International Workshop on Mobile Computing Systems and Applications; February 25–26, 2026; Atlanta, GA ISBN: 978-1-4503-XXXX-X/2018/06 97 CCS: Computing methodologies Mixed / augmented reality CCS: Human-centered computing Ubiquitous and mobile computing systems and tools Yiqin Zhao, Tian Guo Abstract . Mobile sensing systems have long faced a fundamental trade-off between sensing quality and efficiency due to constraints in computation, power, and other limitations. Sparse sensing, which aims to acquire and process only a subset of sensor data, has been a key strategy for maintaining performance under such constraints. However, existing sparse sensing methods often suffer from reduced accuracy, as missing information across space and time introduces uncertainty into many sensing systems. In this work, we investigate whether foundation models can change the landscape of mobile sparse sensing. Using real-world mobile AR data, our evaluations demonstrate that foundation models offer significant improvements in geometry-aware image warping, a central technique for enabling accurate reuse of cross-frame information. Furthermore, our study demonstrates the scalability of foundation model-based sparse sensing and shows its leading performance in 3D scene reconstruction. Collectively, our study reveals critical aspects of the promises and the open challenges of integrating foundation models into mobile sparse sensing systems. 1. Introduction Today’s mobile systems employ sensing systems to understand their surrounding environments to enable rich user experiences. In many mobile sensing systems, including our focus of augmented reality, deep learning models have been a central technology that provides accurate mobile sensing on complex sensor data. However, continuously sensing, often necessary to provide required features for mobile AR applications, can pose a high toll on mobile energy. One natural way is to sense less, which we generally term sparse sensing , but still be able to deliver the same or similar application performance. Traditional sparse sensing techniques reduce system overhead by selectively activating subsets of sensors or processing pipelines ( Yang et al. 2015 ) . However, such systems often struggle to maintain real-world robustness. The advent of foundation models could be a game-changer for sparse sensing. By leveraging their large-scale pretraining and strong generalization capabilities, these models exhibit remarkable robustness in extracting meaningful information even from sparse inputs. For example, an image foundation model can infer omnidirectional environment lighting from just one or a few camera frames ( Zhao et al. 2025 ) . Moreover, recent models, such as DINOv3 ( Siméoni et al. 2025 ) , are capable of producing high-resolution sensing results directly from RGB images with a quality that was previously unattainable even with advanced sensor hardware. In this work, we set out to explore the promises and open challenges when leveraging foundation models for mobile sparse sensing. We focus on mobile augmented reality, which is a fast-growing mobile computing area that deeply depends on multimodal sensing. Specifically, we investigate two research questions: (i) The feasibility of leveraging foundation models on sparse sensing with a data-driven evaluation to quantify the improvement of cross-frame information reuse. (ii) The scalability of sparse sensing on long-duration AR sessions by quantifying the impacts on 3D reconstruction, an important downstream task. To answer the abovementioned research questions, we use an indoor dataset called Scannet++ ( Yeshwanth et al. 2023 ) which consists of real-world mobile AR data recordings. We begin by investigating how foundation models can be leveraged to assist geometry-aware image warping, a central technique for supporting cross-frame information reuse. We tested geometry-aware image warping across different frame intervals, representing different sparsity levels in the temporal domain. In the experiment, we used the device LiDAR depth, the foundation model estimated depth, and the ground truth depth for warping. We measure the error using SSIM on warped RGB and depth images. Our results show that foundation model-based warping significantly outperforms LiDAR-based one, with an average improvement of at least 25.5%. We also investigate the scalability of sparse sensing on long-duration AR sessions and find that foundation model-based 3D reconstruction quality, even with aggressive temporal downsampling, significantly outperforms LiDAR-based reconstruction with 60 FPS sensing. Specifically, using Hausdorff Distance as the metric, foundation model-based reconstruction outperforms LiDAR-based one by 48%. This indicates the possibility of using sparsely sensed data for longer-term environment understanding. Finally, we evaluate the temporal and spatial information differences in AR sessions. We quantify frame-to-frame information overlap under different sparse sensing policies. Our results show that, on average, only about 27% of frames in an AR session are needed to achieve >= 80% information overlap between all consecutive frames. However, none of the traditionally used time interval-based or motion-based control policies can achieve comparable performance. This leaves open questions and challenges for future sparse sensing control policy design. Our observations also point toward a new type of sparse sensing policy design that considers both temporal and spatial dimensions. In summary, we make the following key contributions: • We demonstrate the opportunities to apply sparse sensing in mobile AR with an analysis of how different depth estimation methods impact geometry-aware image warping. We show that foundation models can substantially enhance the accuracy of reusing information in real-time AR sessions. • We show the feasibility of leveraging foundation models to improve the information reuse accuracy on temporally adjacent frames, which could, in turn, allow the use of sparser sensing compared to not using foundation models. • We show that with the help of foundation models, we can achieve comparable or even better 3D reconstruction quality over long-duration AR sessions when compared to sensing with a much higher frequency with a LiDAR sensor, i.e., 15FPS vs. 60FPS. 2. Background Sparse sensing. Sparse sensing aims to achieve accurate sensing from sensor data that contains limited information. Traditional sparse-sensing research often focuses on inferring information from compressed, undersampled, or partial signals to reconstruct rich results from fewer measurements ( Donoho 2006 ) . On mobile platforms, sparse sensing is often used as a strategy to reduce mobile system resource usage by using only a small subset of sensor measurements or computation resources. Existing systems often employ sparse sensing by carefully designing control algorithms and systems to selectively activate sensors or processing frames based on task importance or environmental dynamics ( Razzaque and Dobson 2014 ; Chen et al. 2015 ) . However, sparse inputs often lead to degraded estimation quality since many vision or sensing algorithms rely on dense, temporally consistent data. To address this, prior work ( Yang et al. 2015 ; Duarte-Carvajalino and Sapiro 2009 ) explores adaptive sampling and predictive sensing to balance efficiency and accuracy, yet these approaches remain limited by their task-specific heuristics. Foundation model. Foundation models are large-sized and multi-task-capable models pretrained on diverse datasets. They offer strong generalization and robustness for several tasks. The emergence of foundation models has transformed tasks across computer vision, natural language processing, and robotics. In vision, image encoders such as DINOv3 ( Siméoni et al. 2025 ) and SAM ( Ravi et al. 2024 ) demonstrate strong zero-shot segmentation and object recognition capabilities. In multimodal learning, models like FLAVA ( Singh et al. 2022 ) and Florence ( Yuan et al. 2021 ) integrate text, vision, and geometry, enabling joint reasoning over heterogeneous sensory inputs. Unlike traditional task-specific models that are prone to overfitting,