3D Gaussian and Diffusion-Based Gaze Redirection

Abiram Panchalingam Indu P. Bodala Stuart E. Middleton [email protected] [email protected] [email protected] School of Electronics and Computer Science, University of Southampton · 2025

Read 3D Gaussian and Diffusion-Based Gaze Redirection by Abiram Panchalingam Indu P. Bodala Stuart E. Middleton [email protected] [email protected] [email protected] School of Electronics and Computer Science, University of Southampton (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-11231.txt.

Introduction

Abiram Panchalingam, Indu Bodala, and Stuart Middleton present DiT-Gaze, a framework that redirects a person's gaze in images with high fidelity. Redirected gaze data help gaze-tracking software generalize, but current 3D splatting models struggle with subtle, continuous shifts of the eyes. Small errors in angle can spoil the training value of synthetic faces.

DiT-Gaze pairs a diffusion transformer for sharper synthesis with weak supervision across synthetic middle gaze angles, giving training a smooth sweep of directions. A constraint loss keeps the internal codes for gaze, head pose, and expression separate. Tests set a new best in quality and accuracy, cutting gaze error by about four percent.

Catalog record

CREATORAbiram Panchalingam Indu P. Bodala Stuart E. Middleton [email protected] [email protected] [email protected] School of Electronics and Computer Science, University of Southampton
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-11231

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

3D Gaussian and Diffusion-Based Gaze Redirection Abiram Panchalingam Indu P. Bodala Stuart E. Middleton [email protected] [email protected] [email protected] School of Electronics and Computer Science, University of Southampton Abstract High-fidelity gaze redirection is critical for generating augmented data to improve the generalization of gaze estimators. 3D Gaussian Splatting (3DGS) models like GazeGaussian represent the state-of-the-art but can struggle with rendering subtle, continuous gaze shifts. In this paper, we propose DiT-Gaze, a framework that enhances 3D gaze redirection models using a novel combination of Diffusion Transformer (DiT), weak supervision across gaze angles, and an orthogonality constraint loss. DiT allows higher-fidelity image synthesis, while our weak supervision strategy using synthetically generated intermediate gaze angles provides a smooth manifold of gaze directions during training. The orthogonality constraint loss mathematically enforces the disentanglement of internal representations for gaze, head pose, and expression. Comprehensive experiments show that DiT-Gaze sets a new state-of-the-art in both perceptual quality and redirection accuracy, reducing the state-of-the-art gaze error by 4.1% to 6.353 degrees, providing a superior method for creating synthetic training data. Our code and models will be made available for the research community to benchmark against. 1 Introduction Gaze redirection is a critical task for augmenting datasets to improve the generalization of gaze estimators, which often fail in cross-domain scenarios when encountering out-of-distribution data. This process, which involves manipulating a person’s gaze in an image while preserving their identity, generates synthetic data that can enhance the robustness of these estimators. Early works in this domain formulated gaze redirection as a 2D image-to-image translation problem, relying on techniques like image warping 3 or generative models such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) 4 ; 30 . However, these 2D methods fundamentally overlooked the 3D nature of head and eye movements, often resulting in poor spatial consistency, visual artifacts, and a limited range of redirection. Figure 1 : Gaze redirection: Given an input image and target gaze, DiTGaze utilizes a 3DGS model and a DiT renderer to generate high-fidelity head images with accurate gaze redirection. To address these limitations, the task was reformulated as a 3D-aware problem, leveraging advancements in neural rendering. GazeNeRF 19 was a pioneering method that used Neural Radiance Fields (NeRF) 11 to model the face. It introduced a two-stream MLP architecture to separately represent the face and eye regions, enabling a 3D rotation to be applied to the volumetric eye features before rendering. While this approach significantly improved 3D consistency, GazeNeRF and other NeRF-based methods are hindered by the high computational demands and slow rendering speeds inherent to volumetric rendering. The current state-of-the-art, GazeGaussian 24 , advanced the field by being the first to leverage 3D Gaussian Splatting (3DGS) 8 for this task, overcoming the speed limitations of NeRF. By adopting a similar two-stream model and introducing a novel eye rotation field for explicit control, it achieves higher redirection accuracy and significantly faster rendering speeds than its NeRF-based predecessors. However, despite its success, its U-Net-based renderer presents an opportunity for improvement with a more powerful generative architecture, and like many models, it can struggle to synthesize the subtle, intermediate gaze angles not explicitly seen in the discrete training pairs. In this paper, we propose DiT-Gaze, a 3D Gaussian Splatting-based framework with two key contributions. First, we use the Diffusion Transformer (DiT) architecture for the first time in the field of gaze redirection, leveraging the superior scalability and self-attention mechanism of transformers for higher-fidelity image synthesis 16 . In addition, to render a smooth manifold of gaze redirection, we introduce a weak supervision strategy directly in the 3D Gaussian space. Second, we implement a novel Orthogonality Constraint Loss to mathematically enforce the disentanglement of the internal representations for gaze, head pose, and expression, a technique not previously applied to 3DGS-based avatars. 2 Related Work 2.1 Gaze Redirection Early approaches formulated gaze redirection as a 2D image manipulation task 3 ; 22 . These methods included image warping techniques such as DeepWarp 3 , which were limited in their ability to handle large gaze shifts, and generative models like conditional GANs 4 ; 2 , which improved image quality but fundamentally lacked 3D awareness. Other methods such as STED 30 and ReDirTrans 7 also operated in 2D by applying rotations in a learned latent space, often resulting in poor spatial consistency and visual artifacts. The limitations of 2D manipulation prompted a shift toward 3D-aware solutions. GazeNeRF 19 led this transition by employing Neural Radiance Fields (NeRF). It introduced a critical architectural principle: a two-stream model that decouples the representation of the static face from the dynamic eye regions. This separation was designed to achieve disentangled control, allowing the gaze to be manipulated independently of the head pose. In this framework, gaze redirection is performed by applying a rigid 3D rotation to the eye region’s neural field before compositing it with the face field. Despite its improved consistency, the method’s reliance on NeRF’s volumetric rendering resulted in significant computational overhead and slow inference. Moreover, this feature-level manipulation can still struggle to preserve identity and fine-grained textures, often producing soft or blurred results. GazeGaussian 24 represents the state-of-the-art by being the first to leverage 3D Gaussian Splatting (3DGS) 8 for this task, overcoming the speed limitations of NeRF. Adopting a similar two-stream model, GazeGaussian introduces a novel Eye Rotation field that explicitly adjusts the positions of the 3D eye Gaussians to simulate a rigid rotation. This explicit control mechanism, combined with the efficiency of 3DGS, allows it to achieve higher redirection accuracy and significantly faster rendering speeds than its NeRF-based predecessors. 2.2 3D Head Avatar Synthesis The synthesis of dynamic 3D head avatars is a foundational area of research that provides the underlying technology for 3D-aware gaze redirection. Early work in this domain utilized parametric 3D head models like FLAME 1 to map expression and pose parameters directly to 3D facial geometry. More recent work has shifted to neural rendering techniques, which can be broadly categorized into NeRF-based and 3DGS-based methods. One major approach uses Neural Radiance Fields (NeRF) to create high-fidelity, controllable avatars. Models such as HeadNeRF 6 leverage neural radiance fields to deform facial movements from a canonical space, conditioned on parameters for shape, expression, and lighting. However, these methods are often limited by the high computational demands and slow rendering speeds inherent to volumetric rendering. More recently, 3D Gaussian Splatting (3DGS) has emerged as a superior alternative, offering impressive rendering quality at significantly faster speeds. State-of-the-art models like Gaussian Head Avatar 26 initialize 3D Gaussians from a neutral mesh and use MLPs to deform them, creating ultra high-fidelity dynamic avatars. While these general head avatar models are powerful for facial animation, they often neglect the mechanisms for precise gaze control, a key limitation that specialized models like GazeGaussian are designed to address. 2.3 Diffusion Models for Image Synthesis Diffusion models have become the state-of-the-art for high-fidelity image synthesis, operating through a forward process that gradually adds noise to data and a learned reverse process that iteratively denoises it to generate a clean sample 5 . Early diffusion models often operated directly in the pixel space, using convolutional U-Net backbones 18 to predict the noise component at each step of the reverse process. To improve computational efficiency,

Read the complete text