2D-3D Attention and Entropy for Pose Robust 2D Facial Recognition

J. Brennan Peace, Shuowen Hu, and Benjamin S. Riggan IMAGES Lab., ECE Dept., University of Nebraska-Lincoln, Lincoln, Nebraska, USA U.S. Army Research Laboratory, Adelphi, Maryland, USA · 2025

Read 2D-3D Attention and Entropy for Pose Robust 2D Facial Recognition by J. Brennan Peace, Shuowen Hu, and Benjamin S. Riggan IMAGES Lab., ECE Dept., University of Nebraska-Lincoln, Lincoln, Nebraska, USA U.S. Army Research Laboratory, Adelphi, Maryland, USA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-09073.txt.

Introduction

Brennan Peace and co-authors taught face recognition in 2025 to borrow depth sense from three dimensional face data, so a flat photo taken in profile can still match a frontal enrollment shot. Big pose gaps remain a stubborn weakness of face systems.

Their framework pairs each photo representation with a point cloud one and forces them to agree. A joint attention map highlights the patterns the two views share, and a joint entropy term rewards consistent pairings. On two test sets with hard profile views, the method beat rivals on strict false alarm settings by clear margins. The authors pitch it for security screening where the camera angle cannot be controlled.

Catalog record

CREATORJ. Brennan Peace, Shuowen Hu, and Benjamin S. Riggan IMAGES Lab., ECE Dept., University of Nebraska-Lincoln, Lincoln, Nebraska, USA U.S. Army Research Laboratory, Adelphi, Maryland, USA
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-09073

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

2D-3D Attention and Entropy for Pose Robust 2D Facial Recognition J. Brennan Peace, Shuowen Hu, and Benjamin S. Riggan IMAGES Lab., ECE Dept., University of Nebraska-Lincoln, Lincoln, Nebraska, USA U.S. Army Research Laboratory, Adelphi, Maryland, USA Abstract Despite recent advances in facial recognition, there remains a fundamental issue concerning degradations in performance due to substantial perspective (pose) differences between enrollment and query (probe) imagery. Therefore, we propose a novel domain adaptive framework to facilitate improved performances across large discrepancies in pose by enabling image-based (2D) representations to infer properties of inherently pose invariant point cloud (3D) representations. Specifically, our proposed framework achieves better pose invariance by using (1) a shared (joint) attention mapping to emphasize common patterns that are most correlated between 2D facial images and 3D facial data and (2) a joint entropy regularizing loss to promote better consistency—enhancing correlations among the intersecting 2D and 3D representations—by leveraging both attention maps. This framework is evaluated on FaceScape and ARL-VTF datasets, where it outperforms competitive methods by achieving profile ( 90 ∘ + 90^{\circ}+ ) TAR @ 1% FAR improvements of at least 7.1% and 1.57%, respectively. I Introduction Over the last decade, facial recognition (FR) systems have significantly advanced in terms of recognition—verification (1:1 matching) and/or identification (1:N matching)—performance. Despite overall FR enhancements across progressively increasing variations in pose, illumination, expression, occlusion, resolution, and scale, FR systems often under-perform [ 18 , 17 , 58 ] , when matching gallery and query (probe) images with relatively large perspective differences (i.e., greater than 60 degrees). Therefore, FR systems rarely manifest properties of pose invariance, which are ideal for unconstrained recognition and non-cooperative enrollment. Our motivation is to address this critical challenge by leveraging 3D priors—probability distributions representing assumptions about parametric models before incorporating specific observations—to bridge the gap between 2D and 3D representations to improve robustness to larger pose variations. Fig. 1 : Conventional 2D image embedding networks for FR often exhibit mismatches due to not being invariant to pose. Instead, we introduce a framework that implicitly infers pose invariant 3D information when extracting 2D image embeddings via our proposed 2D-3D Joint Attention Mapping (JAM) and 2D-3D Joint Entropy regularizing loss. This framework processes 2D images (left) to output image representations (right) that are inherently robust to pose variations. For example, images A and B yield similar representations from the same identity, while B and D yield distinct ones despite a similar pose, as they depict different identities. Subjects consented for publication. Although some FR applications, such as user authentication or airport kiosks, are able to acquire images with optimal frontal viewing angles (pose) and controlled conditions (e.g., lighting and resolution), many government—defense, intelligence, and law enforcement—FR systems operate under less than optimal conditions, such as using off-pose imagery. In operational settings where subjects are not controlled and are non-cooperative, there are significant benefits with addressing the fundamental pose invariance problem for FR. Therefore, the primary objective of this paper is to increase robustness to pose variations by enabling image-based (2D) representations to infer properties of point cloud (3D) representations, which acts as a form of pose invariant guidance (see Figure 1 ). Despite advances in unconstrained FR, such as methods that reduce intra-class variance (e.g., Center Loss [ 67 ] , ArcFace [ 13 ] ), use synthetic views [ 47 ] , and frontalize faces [ 22 ] , 2D imagery still lacks sufficient understanding of underlying pose invariant 3D information—shape, depth, and texture—due to projective camera geometry. To address pose differences, recent literature has employed domain adaptation to model the complex interrelationships among 2D and 3D representations using semantic segmentation [ 46 , 63 ] and 3D priors [ 27 ] . Unlike semantic segmentation methods that directly model relationships among 2D and 3D data, our proposed framework indirectly establishes consistent representations between domains by aligning and isolating common key discriminative information via 3D priors for 2D image classification to improve robustness to a wide range of pose variations, including profile-to-frontal matching. Similar to Pri3D [ 27 ] , our method also incorporates 3D priors to benefit 2D image classification. However, instead of directly building 3D priors from rich light detection and ranging (LiDAR) data, our proposed framework enables 2D representation that indirectly infer properties of pose invariance from 3D morphable models (3DMMs) [ 7 ] of faces. Therefore, a new framework is proposed to enhance 2D FR by jointly learning to focus on both spatial and channel information that permits interoperability between 2D and 3D facial analysis. The learned relationships between 2D and 3D representations are enabled by an implicitly learned mapping between the two facial geometries. This framework significantly differs from existing domain adaptation methods that directly adapt RGB images to depth maps [ 64 ] , which are not pose invariant like point clouds. Additionally, our proposed framework does not rely on the biasing and limiting effects from explicitly masking 3D points that are occluded in corresponding 2D images [ 78 ] . Our framework employs learned 3D priors from point-clouds, distinctively utilizing them to improve 2D FR capabilities, especially when enrollment (gallery) and query (probe) imagery are acquired from extremely different perspectives. This framework is enabled by a domain adaptive optimization scheme that permits 2D representations during inference without the direct use of 3D input data. The primary contributions of this work include: 1. a novel 2D-3D pose invariant FR framework —a domain adaptation strategy that facilitates pose invariance for 2D images via learned 3D priors, 2. a new 2D-3D Joint Attention Mapping (JAM) —a mapping that uses shared attention to modify the spatial and channel information for both 2D and 3D data, 3. a new 2D-3D Joint Entropy (JE) regularization —a constraint that indirectly promotes 2D and 3D embedding networks to have consistent representations by optimizing the joint attention maps. The proposed framework is extensively evaluated across a wide range of pose variations. Not only is the framework evaluated using FaceScape [ 74 ] —a dataset containing 2D and 3D data for training purposes—but the trained model is also evaluated for generalization using a large-scale FR dataset (ARL-VTF [ 53 ] ) that only contains 2D images with significant pose variation. Our approach achieves a notable improvement in pose robust performance, surpassing competitive methods with at least 7.1% and 1.57% TAR @ 1% FAR gains for extreme profile views in FaceScape and ARL-VTF, respectively, demonstrating its effectiveness in unconstrained settings. II Relevant Work 2D FR : Over the last decade, 2D FR has significantly advanced due to improved tools/resources (e.g., hardware, application programming interfaces, and deep learning) and availability of large-scale datasets. DeepFace [ 61 ] and VGG-Face [ 54 ] used convolutional neural networks (CNNs) to identify faces, which improved performance under progressively more challenging resolution/quality, pose, illumination, and expression conditions. FaceNet [ 57 ] enhanced FR by mapping facial images into a compact Euclidean space. SphereFace [ 43 ] introduced an angular margin loss to improve discrimination. Instead, CosFace [ 65 ] optimized the cosine margin, and ArcFace [ 13 ] enhanced FR with a tighter angular margin between classes. Pose-invariant methods such as those by Cao et al. [ 9 ] , Yin et al. [ 75 ] ,

Read the complete text