GA3CE: Unconstrained 3D Gaze Estimation with Gaze-Aware 3D Context Encoding
Yuki Kawana, Shintaro Shiba, Quan Kong, Norimasa Kobori · 2025
Read GA3CE: Unconstrained 3D Gaze Estimation with Gaze-Aware 3D Context Encoding by Yuki Kawana, Shintaro Shiba, Quan Kong, Norimasa Kobori (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-10671.txt.
Introduction
Yuki Kawana, Shintaro Shiba, Quan Kong, and Norimasa Kobori present a 2025 paper on estimating where a person is looking in three dimensions. Most gaze tools need a close view of the eyes. That fails when a person is far away or turned aside, exactly the cases in many real scenes.
Their method, GA3CE, reads the 3D layout around the subject: body pose and object positions form a context. That context is recentered on the subject's own viewpoint, which shrinks the spatial puzzle. A new positional code splits direction from distance to sharpen the geometry. Tests on benchmark data cut the mean angle error by thirteen to thirty-seven percent against leading baselines. Spatial reasoning, not just eye pixels, drives the gain.
Catalog record
| CREATOR | Yuki Kawana, Shintaro Shiba, Quan Kong, Norimasa Kobori |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-10671
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GA3CE: Unconstrained 3D Gaze Estimation with Gaze-Aware 3D Context Encoding Yuki Kawana, Shintaro Shiba, Quan Kong, Norimasa Kobori Abstract We propose a novel 3D gaze estimation approach that learns spatial relationships between the subject and objects in the scene, and outputs 3D gaze direction. Our method targets unconstrained settings, including cases where close-up views of the subject’s eyes are unavailable, such as when the subject is distant or facing away. Previous approaches typically rely on either 2D appearance alone or incorporate limited spatial cues using depth maps in the non-learnable post-processing step. Estimating 3D gaze direction from 2D observations in these scenarios is challenging; variations in subject pose, scene layout, and gaze direction, combined with differing camera poses, yield diverse 2D appearances and 3D gaze directions even when targeting the same 3D scene. To address this issue, we propose GA3CE: Gaze-Aware 3D Context Encoding. Our method represents subject and scene using 3D poses and object positions, treating them as 3D context to learn spatial relationships in 3D space. Inspired by human vision, we align this context in an egocentric space, significantly reducing spatial complexity. Furthermore, we propose D 3 (direction-distance-decomposed) positional encoding to better capture the spatial relationship between 3D context and gaze direction in direction and distance space. Experiments demonstrate substantial improvements, reducing mean angle error by 13%–37% compared to leading baselines on benchmark datasets in single-frame settings. Project page: woven-visionai.github.io/ga3ce-project Figure 1 : (a) Our method estimates 3D gaze direction from an RGBD image and camera intrinsics. (b) Prior work [ 16 ] estimates gaze from 2D representations, incorporating 3D spatial cues only as a non-learnable post-processing step. (c) Since direct 3D gaze estimation from 2D representations is challenging, we use 3D pose and object positions as intermediate representations. (d) We introduce gaze-aware 3D context encoding (GA3CE), transforming the 3D context into a unified directional space. (d1) This space aligns with a egocentric (subject-centered) view, normalized relative to the head position and oriented to the view direction. (d2) This alignment enables the decomposition of a 3D point into direction and distance in egocentric space, with D 3 positional encoding capturing their correlations. (e) The transformer then learns spatial relationships between the subject, objects, and 3D gaze. 1 Introduction Gaze direction is a powerful non-verbal cue for understanding human engagement, interest, and attention. Humans can often infer another person’s gaze direction based on various appearance cues, even when clear, close-up views of the eyes are not available, such as when the person is distant or facing away. This is possible because humans interpret gaze through cues like body pose and movement, scene context, and spatial relationships. Estimating 3D gaze direction in such unconstrained settings has a range of applications. Consider scenarios where wearable eye-tracking devices are impractical, such as detecting whether a pedestrian is attentive to traffic via a surveillance camera [ 22 , 2 ] or analyzing customer engagement in a retail environment using video monitoring [ 60 , 59 ] . To estimate 3D gaze direction in these unconstrained settings, previous work has utilized cues that imply gaze direction, such as a subject’s temporal 3D direction [ 40 ] . Recently, depth maps have been incorporated to analytically constrain 3D gaze direction estimated from head appearance [ 16 ] . However, previous works overlook spatial relationships between the subject, scene, and gaze direction; scene context is not considered [ 40 ] or only considered during non-learnable, analytical step [ 16 ] . Therefore, an effective learning approach that holistically considers both subject and scene for 3D gaze estimation remains largely unexplored. In this work, we propose a novel 3D gaze estimation approach that learns to understand the spatial relationship between the scene, subject, and gaze. Following the closest previous work [ 16 ] , our method estimates 3D gaze direction given an RGB image, a depth map of the scene from a sensor or a zero-shot estimator, and camera intrinsics. In contrast to the gaze-following task which focuses on detecting the visible gazed point , our focus is to estimate the 3D direction of gaze. The task has orthogonal benefit to the gaze-following task, as it can output gaze information even when an attended point is not visible due to occlusion or being out of sight. Estimating 3D gaze direction from 2D observations is challenging due to variations in 2D scene context and gaze direction. Even with identical 3D scenes, 2D appearances of the subject and scene, and ground truth gaze direction defined in camera space, change significantly with different camera poses, as each pose projects the 3D world differently onto a 2D image. This creates complex interactions between the relative positions of 2D features and 3D gaze directions across varying camera poses. Normalizing close-up 2D facial images with perspective correction to reduce appearance variation from different camera poses has been widely studied for 3D gaze estimation [ 52 , 71 ] . However, the normalization beyond the subject’s 2D face appearance remains largely unexplored. To address the complexity of 3D gaze estimation, we focus on three key challenges in this paper: ( i ) What is an effective representation of the subject and scene? ( ii ) How can variations in subject, scene, and gaze due to different camera poses be normalized? ( iii ) How can we model the spatial relationships among subject, scene, and gaze? For ( i ), we represent the subject as 3D keypoints and the scene as 3D points corresponding to object positions. Instead of estimating 3D gaze direction directly from the 2D appearance of the subject and scene, these serve as robust intermediate 3D context representations. For ( ii ) and ( iii ), inspired by human vision studies [ 38 , 7 , 9 ] demonstrating that direction and distance to objects in the scene within subject’s view strongly influences gaze, we propose GA3CE , for Gaze-Aware 3D Context Encoding. To address ( ii ), this intermediate 3D representation enables geometric transformation of 3D contexts relative to the subject’s view direction, such as prior gaze or head direction. This transformation as normalization reduces context variations across camera poses, creating a unified, egocentric space. To address ( iii ), we decompose the 3D context into direction and distance components for positional encoding, termed D 3 positional encoding, which better captures positional and directional similarities. This decomposition, combined with a transformer-based architecture, models the spatial interactions between 3D context and the final 3D gaze direction. An overview of our method is shown in Fig. 1 . Experiments on three benchmark datasets demonstrate that our approach significantly outperforms previous methods, and an ablation study highlights the advantages of using a 3D representation with GA3CE for learning spatial relationships. Our contributions are summarized as follows: 1. We propose a novel approach for 3D gaze estimation based on an explicit understanding of the spatial relationships between a subject and objects in a 3D context. 2. We propose GA3CE , for Gaze-Aware 3D Context Encoding, to enhance the representation of the relationship between 3D context and gaze. 3. We achieve state-of-the-art quantitative and qualitative performance on three benchmark datasets [ 16 , 40 , 23 ] . Figure 2 : Pipeline overview. PE = positional