GazePrior: Zero-Shot AR/VR Eye Tracking via Learned 3D Gaze Reconstruction

Corentin Dumery, David Colmenares, Alexander Fix, Pascal Fua, Ali Behrooz, Jogendra Kundu · 2026

Read GazePrior: Zero-Shot AR/VR Eye Tracking via Learned 3D Gaze Reconstruction by Corentin Dumery, David Colmenares, Alexander Fix, Pascal Fua, Ali Behrooz, Jogendra Kundu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22359.txt.

Introduction

Dumery and colleagues present GazePrior, a learned 3D prior for eye tracking in augmented and virtual reality. Training a tracker for each new device is costly, and synthetic data often lacks realism. GazePrior models how human eyes vary across identities, gaze directions and light settings. It rebuilds annotated data from older devices in 3D from sparse inputs. That data can then be rendered from any target device's cameras. Trackers trained on the synthetic data beat prior zero-shot methods on accuracy and robustness.

Catalog record

CREATORCorentin Dumery, David Colmenares, Alexander Fix, Pascal Fua, Ali Behrooz, Jogendra Kundu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52240
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.22359

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GazePrior: Zero-Shot AR/VR Eye Tracking via Learned 3D Gaze Reconstruction Corentin Dumery, David Colmenares, Alexander Fix, Pascal Fua, Ali Behrooz, Jogendra Kundu Abstract Eye tracking (ET) is a foundational technology for advanced AR/VR applications. However, training ET models for every new device is challenging: real data collection is costly and time-consuming, while existing synthetic data generation methods lack realism. To remove the need for additional data collection while maintaining data quality, we introduce a data-driven 3D prior that models the distribution of human eyes across diverse identities, gaze directions, and light settings. This model, which we coin GazePrior , then enables sparse-input 3D reconstruction of annotated data collected with previous ET devices, which can in turn be rendered from the cameras of any target ET device. Our approach synthesizes data with the realism, diversity, and ground-truth accuracy of real data collection without its prohibitive costs. Our experiments demonstrate that ET models trained with our synthesized data outperform previous zero-shot methods, achieving higher accuracy and robustness. More information is available on the project page: https://corentindumery.github.io/projects/gazeprior.html . † † affiliation: Meta Reality Labs, Redmond, USA † † affiliation: EPFL, Lausanne, Switzerland † † contribution: Work done during an internship at Meta † † Keywords: Eye Tracking, Data Synthesis, 3D Reconstruction, Shape Prior, Sparse-Input Reconstruction † † correspondence: Jogendra Kundu at [email protected] 1 Introduction Eye tracking is key to the next-generation of smart glasses and AR/VR headsets. Typical devices [ 24 , 51 , 52 , 25 , 14 ] rely on up to 5 cameras per eye to reliably and accurately estimate gaze direction and 3D pupil location, with a required precision of the order of a few degrees to enable essential features such as foveated rendering and gaze-based interaction. The captured images are fed to deep networks that have to be properly trained. In theory, this is a simple matter of collecting appropriate training sets. However, in industrial practice, this is an expensive and time-consuming affair because the hardware configuration of the devices, including camera placement and image resolution, evolve over time, requiring the acquisition of new training sets comprising millions of frames with high-quality ground truth for each new device generation. In this paper, we propose leveraging the existence of large-scale datasets with ground-truth annotations that were collected while developing earlier headsets. Our goal is to repurpose this data to train eye tracking (ET) networks that can handle the images acquired by newer headsets featuring different camera setups without having to acquire any images using the new device, in other words, a zero-shot approach to eye tracking in AR/VR. To this end, we propose a semi-synthetic approach to generating training images for the new devices given only the ones acquired using existing older configurations and the new camera specification and configuration. This means that networks can be trained for new devices even before they are built, which will enable precise evaluation of how camera placement will impact ET performance. Thus, it holds the promise of accelerating development without incurring the prohibitive costs associated with large-scale data collection. Figure 1: Overview of GazePrior. We learn a 3D eye prior ( top ) to achieve high-fidelity retargeting of annotated eye data ( bottom left ). This enables robust eye tracking on new devices ( bottom right ) without requiring any device-specific data collection. At the heart of our approach is the following insight. Even though capturing different people using various camera configurations results in a great diversity of images, there are nevertheless strong similarities across subjects, gaze directions, and lighting environments. Inspired by recent 3D avatar approaches [ 8 , 7 ] , we model this prior as a conditional radiance field, which ensures that the generated views maintain spatial and photometric consistency across varying viewpoints. These methods achieve high-quality avatar reconstruction from only sparse input views by learning a multi-subject latent-vector representation and leveraging it to condition 3D reconstruction. In this work, we extend this latent-vector approach by conditioning on identity, gaze, and illumination in higher-dimensional latent spaces, and use it to synthesize new training images for a new setup from existing ones. We opt for conditional neural radiance fields for this purpose because they demonstrate superior accuracy across recent benchmarks [ 35 ] and are well-suited for conditioning on high-dimensional latent embeddings. This contrasts with earlier semi-synthetic approaches to eye image synthesis [ 67 , 42 , 43 ] that reconstruct each new eye from scratch. Such methods require dense input views unavailable in standard ET setups, drastically reducing the diversity and quantity of data they can use. To compensate, they often synthesize new gaze directions for the few available subjects. A typical approach is to rely on explicit eyeball masks to reconstruct the eyeball separately from the face [ 78 , 42 , 67 , 43 ] , then rotate the mesh to simulate new gaze directions. However, this ignores the intricate coupling between eyeballs, eyelids, and other facial features essential for physically-plausible synthesis, degrading accuracy and limiting usefulness when even small gaze errors disrupt immersion. Instead, we argue that this coupling should be learned by a network. In short, state-of-the-art methods for synthesizing eye tracking images have severe limitations that are summarized in Tab. 1 and which our approach removes. Our main contributions are summarized as follows: • We are the first to introduce a learned gaze prior that disentangles subject, gaze direction, and light, with biologically accurate eyelid movements. We do not rely on eyeball rotations that limit the realism of synthesized data. • We incorporate this model into a novel-view-synthesis pipeline that synthesizes images that can be used to train an eye tracker to perform accurately and reliably on real-images. • We demonstrate the applicability of our synthesized data to zero-shot eye tracking, achieving precise gaze prediction and unlocking new capabilities for models without any device-specific data collection. To compare our approach for eye tracking purposes against very recent eye-image synthesis baselines [ 43 , 42 , 78 ] , we rely on the state-of-the-art ET architecture and system of the Aria [ 14 ] headset. When trained on our images instead of the baselines, it consistently achieves superior performance. 2 Related Works Gaze Estimation. Estimating 3D gaze direction from in-the-wild images has been a long-standing computer vision challenge. Early approaches typically relied on geometric features, employing additional cues like face landmarks [ 59 , 44 , 46 , 91 ] , RGB-D images [ 54 , 18 , 17 ] , or explicitly fitting a morphable 3D eye model to the image data [ 80 , 82 , 72 , 34 , 79 ] . The field saw a significant shift with the advent of machine learning and deep learning techniques [ 47 , 49 ] , moving beyond initial linear regression models [ 48 ] . This transition, alongside the creation of the first evaluation benchmarks [ 17 , 90 , 92 ] , highlighted the critical importance of large, high-quality training data. Several strategies emerged to address this data scarcity: crowdsourcing eye tracking data [ 33 ] , augmenting real datasets by generating novel view angles while preserving the original gaze direction [ 72 ] , or creating vast synthetic datasets using rendering

Read the complete text