The Good, the Better, and the Best: Improving the Discriminability of Face Embeddings through Attribute-aware Learning
Ana Dias, João Ribeiro Pinto, Hugo Proença, João C. Neves, [4pt] University of Beira Interior, Portugal · 2026
Read The Good, the Better, and the Best: Improving the Discriminability of Face Embeddings through Attribute-aware Learning by Ana Dias, João Ribeiro Pinto, Hugo Proença, João C. Neves, [4pt] University of Beira Interior, Portugal (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15062.txt.
Introduction
Dias and co-authors improve face embeddings by teaching them with facial attributes, grouped by how much they help identity. Face models still struggle with age, pose, and occlusion changes. The team trains with identity labels plus relevant and non-relevant attributes, kept in clear groups. Tests on standard verification benchmarks give two clear lessons. Identity-relevant attributes help more than a broad attribute set. Forcing embeddings to unlearn non-identity attributes adds further gains. The method also works as a diagnostic for shortcut learning.
Catalog record
| CREATOR | Ana Dias, João Ribeiro Pinto, Hugo Proença, João C. Neves, [4pt] University of Beira Interior, Portugal |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28913 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.15062
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Good, the Better, and the Best: Improving the Discriminability of Face Embeddings through Attribute-aware Learning Ana Dias, João Ribeiro Pinto, Hugo Proença, João C. Neves, [4pt] University of Beira Interior, Portugal Abstract Despite recent advances in face recognition, robust performance remains challenging under large variations in age, pose, and occlusion. A common strategy to address these issues is to guide representation learning with auxiliary supervision from facial attributes, encouraging the visual encoder to focus on identity-relevant regions. However, existing approaches typically rely on heterogeneous and fixed sets of attributes, implicitly assuming equal relevance across attributes. This assumption is suboptimal, as different attributes exhibit varying discriminative power for identity recognition, and some may even introduce harmful biases. In this paper, we propose an attribute-aware face recognition architecture that supervises the learning of facial embeddings using identity class labels, identity-relevant facial attributes, and non-identity-related attributes. Facial attributes are organized into interpretable groups, making it possible to decompose and analyze their individual contributions in a human-understandable manner. Experiments on standard face verification benchmarks demonstrate that joint learning of identity and facial attributes improves the discriminability of face embeddings with two major conclusions: (i) using identity-relevant subsets of facial attributes consistently outperforms supervision with a broader attribute set, and (ii) explicitly forcing embeddings to unlearn non-identity-related attributes yields further performance gains compared to leaving such attributes unsupervised. Additionally, our method serves as a diagnostic tool for assessing the trustworthiness of face recognition encoders by allowing for the measurement of accuracy gains with suppression of non-identity-relevant attributes, with such gains suggesting shortcut learning from redundant attributes associated with each identity. I Introduction (a) Previous Methods (b) Proposed Method Fig. 1: From fixed attribute supervision to selective prediction and suppression. (a) Prior multi-task face recognition methods use a fixed attribute list as an auxiliary prediction task alongside identity learning. (b) Our approach organizes attributes into groups and controls their influence during training: groups can be predicted or selectively suppressed, while the identity learning objective remains unchanged. Despite significant advances over the past decade [ 18 , 4 , 11 ] , face recognition methods still struggle to correctly identify individuals under factors such as age variation [ 7 , 21 ] , pose changes [ 1 , 8 ] , and occlusion [ 15 , 5 ] . To address these limitations, a common strategy is the use of auxiliary supervision during training, usually with facial attributes, to guide representation learning toward identity-related cues. This is typically achieved within a multi-task learning framework, in which a shared visual encoder is jointly optimized for identity recognition and facial attribute estimation [ 16 , 20 , 13 , 14 ] . However, existing multi-task face recognition approaches usually treat facial attributes as a single, homogeneous auxiliary objective defined over a fixed attribute set. This formulation neglects two key aspects: (a) facial attributes differ substantially in their relevance and discriminative power for identity recognition, and some may even introduce noise or bias into the learned representation; and (b) actively suppressing certain attributes can be more beneficial than simply leaving them unsupervised. Motivated by these observations, we introduce a face recognition architecture that enhances the discriminability of identity-related features by encouraging the encoder to preserve information associated with identity-relevant facial attributes while ignoring or suppressing information related to attributes that are irrelevant for recognition (Figure 1 ). To this end, we adopt a state-of-the-art visual encoder supervised by three complementary loss signals: identity recognition, identity-relevant facial attributes, and identity-irrelevant facial attributes. The first two loss signals are jointly optimized using standard gradient descent to promote the learning of identity-discriminative features. In contrast, the loss associated with identity-irrelevant attributes is combined with a gradient reversal mechanism, encouraging the model to explicitly “unlearn” attribute information that is detrimental to identity recognition. Furthermore, facial attributes are organized into interpretable, region-based groups, enabling a systematic and human-understandable analysis of how different attribute subsets contribute to identity learning. Using the proposed architecture, we demonstrate that jointly learning identity and facial attributes is clearly beneficial to identity-only supervision, but, as hypothesized by us, this increment is suboptimal, since identity-relevant attribute groups yields stronger and more stable improvements than using a heterogeneous mix of attributes. Moreover, we show that explicitly suppressing non-identity-related attributes provides even further performance gains, supporting the hypothesis that actively suppressing weakly relevant attributes can be more effective than simply leaving them unsupervised. While most attribute groups support these conclusions, attributes related to hair deviate from the general trend. Despite being non-identity-relevant, its suppression degrades recognition performance, indicating that this region encodes identity-discriminative information. We hypothesize that this is due to the low intra-subject variability of hair-related attributes in common face datasets, and we demonstrate this claim by achieving \approx 80\% verification accuracy solely by using hair-related attributes, without any visual input. These findings expose attribute–identity correlations that are often overlooked in face recognition benchmarks, and demonstrate how our method can be leveraged to identify and analyze such biases. Accordingly, beyond improving recognition performance, the proposed approach provides a useful tool for promoting more transparent and trustworthy face recognition systems by explicitly revealing which attributes contribute to identity discrimination. In summary, our main contributions are as follows: • We introduce an attribute-aware face recognition architecture that jointly learns identity representations and facial attributes, while explicitly suppressing non-identity-related attribute information via a gradient reversal mechanism. • We experimentally demonstrate that supervising selected identity-relevant attribute groups, together with explicit suppression of non-identity attributes, consistently outperforms joint learning identity with a heterogeneous mix of facial attributes. • We show that selective attribute suppression not only improves recognition performance, but also helps expose shortcut learning in attribute-supervised face recognition, evidencing the potential of our approach for analyzing dataset biases and promoting more transparent and trustworthy recognition systems. II State of Art II-A Multi-task Face Recognition Early research on multi-task face recognition explored the use of facial attributes to improve identity learning through auxiliary supervision. Wang et al. [ 20 ] proposed a joint optimization framework for face recognition and attribute prediction and reported improved recognition performance, while Taherkhani et al. [ 16 ] showed that attributes provide complementary information for identity recognition. Zhong et al. [ 24 ] further showed that face recognition representations capture attribute-related information, consistent with the prior benefits of attribute supervision. More recent frameworks extend this idea to multi-task face perception models with shared backbones and task-specific heads, jointly training face recognition with tasks such as attribute prediction, age estimation, and expression recognition [ 14 , 13 ] . While these methods demonstrate the effectiveness of joint learning, they typically adopt a fixed attribute list and treat attribute supervision as a single auxiliary objective. Consequently, the effects of different attribute groups, and their selective prediction or suppression, on face recognition performance remain underexplored. Our multitask learning architecture enables controlled prediction and suppression of selected attribute groups and explores their effects on face recognition performance. II-B Adversarial Learning for Invariance Gradient Reversal Layers (GRLs) [ 6 ] have been adopted in multi-task face recognition to suppress attribute-specific information, most commonly age, from identity representations by reversing the gradient from an auxiliary branch during backpropagation. MTLFace [ 7 ] integrates a recognition discriminator with a