AttriBE: Quantifying Attribute Expressivity in Body Embeddings for Recognition and Identification

Basudha Pal, Siyuan Huang, Anirudh Nanduri, Zhaoyang Wang, and Rama Chellappa · 2026

Read AttriBE: Quantifying Attribute Expressivity in Body Embeddings for Recognition and Identification by Basudha Pal, Siyuan Huang, Anirudh Nanduri, Zhaoyang Wang, and Rama Chellappa (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27218.txt.

Introduction

Pal and four colleagues extend expressivity, the mutual information between learned features and a given attribute, to person re-identification across images and video. Existing systems match people across frames but are swayed by gender, pose, and body mass index, raising fairness and generalization worries. Using a second neural network to measure encoding strength, they test three transformer-based models on a large visible-spectrum dataset. Body mass index shows the highest expressivity in deeper layers, ranked above pitch, gender, and yaw.

Catalog record

CREATORBasudha Pal, Siyuan Huang, Anirudh Nanduri, Zhaoyang Wang, and Rama Chellappa
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63353
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.27218

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AttriBE: Quantifying Attribute Expressivity in Body Embeddings for Recognition and Identification Basudha Pal, Siyuan Huang, Anirudh Nanduri, Zhaoyang Wang, and Rama Chellappa Abstract Person re-identification (ReID) systems that match individuals across images or video frames are essential in many real-world applications. However, existing methods are often influenced by attributes such as gender, pose, and body mass index (BMI), which vary in unconstrained settings and raise concerns related to fairness and generalization. To address this, we extend the notion of expressivity , defined as the mutual information between learned features and specific attributes, using a secondary neural network to quantify how strongly attributes are encoded. Applying this framework to three transformer-based ReID models on a large-scale visible-spectrum dataset, we find that BMI consistently shows the highest expressivity in deeper layers. Attributes in the final representation are ranked as BMI > Pitch > Gender > Yaw \text{BMI}>\text{Pitch}>\text{Gender}>\text{Yaw} , and expressivity evolves across layers and training epochs, with pose peaking in intermediate layers and BMI strengthening with depth. We further extend the analysis to cross-spectral person identification across infrared modalities including short-wave, medium-wave, and long-wave infrared. In this setting, pitch becomes comparable to BMI and attribute trends increase monotonically across depth, suggesting increased reliance on structural cues when bridging modality gaps. Overall, the results show that transformer-based ReID embeddings encode a hierarchy of implicit attributes, with morphometric information persistently embedded and pose contributing more strongly under cross-spectral conditions. I Introduction † † footnotetext: This article extends our 2025 IEEE International Joint Conference on Biometrics (IJCB) paper with new cross-spectral analysis and expanded theoretical validation. Deep neural networks trained for biometric recognition are optimized to encode identity-discriminative information, yet they frequently capture unintended latent attributes that are not explicitly supervised. In face recognition, identity embeddings have been shown to cluster according to demographic variables such as gender while also encoding nuisance factors including pose, age, and illumination [ 21 , 35 , 43 ] . Such latent encoding can influence recognition accuracy and operational behavior under varying capture conditions [ 17 , 30 ] . Systematically characterizing which auxiliary attributes are embedded in learned representations, and the strength with which they are encoded, is therefore essential for interpretability, robustness analysis, and bias diagnosis in biometric systems. To address this question, Dhar et al. introduced the notion of expressivity for face recognition, quantifying the relationship between deep network features and semantic attributes [ 13 ] . Building on this concept, we extend expressivity analysis to person re-identification (ReID) with the goal of understanding how body-related attributes are represented within models trained primarily for identity matching. Person ReID plays a central role in smart-city infrastructure, surveillance analytics, and public safety [ 2 , 26 ] , as well as in autonomous perception systems for pedestrian detection and tracking [ 4 , 53 ] . The task requires reliable identity matching across non-overlapping camera views under substantial variations in pose, appearance, and environment [ 59 ] . Significant advances have been achieved through both image-based and video-based modeling, including temporal aggregation and robustness to clothing or appearance variation [ 20 , 19 ] . Despite these advances, modern ReID systems can unintentionally encode auxiliary body attributes beyond identity, motivating principled post-hoc analysis of internal feature representations. Recent work has demonstrated the presence of attribute information in body-recognition embeddings using downstream probing techniques such as logistic regression [ 33 ] . However, probing accuracy reflects only predictability and provides an indirect proxy for representational dependence, potentially missing nonlinear or non-Gaussian relationships. In contrast, information-theoretic analysis enables direct quantification of statistical dependency between learned features and attributes. Accordingly, we adopt Mutual Information Neural Estimation (MINE) [ 3 ] , grounded in classical information theory [ 12 ] , to measure how much attribute-related information is intrinsically encoded within deep biometric representations. Our prior study accepted at the International Joint Conference on Biometrics (IJCB) introduced this information-theoretic expressivity framework for transformer-based person ReID and analyzed three state-of-the-art architectures namely, SemReID, DCFormer, and PFD, using visible-spectrum imagery from the Broad Range Identification at Altitude and Range (BRIAR) dataset [ 40 , 11 ] . That work provided, to our knowledge, the first systematic quantification of body-attribute encoding in deep person-recognition embeddings through mutual information (MI) analysis. However, the investigation was restricted to a single sensing modality and did not examine how attribute encoding behaves under spectral domain shift. In operational biometric environments, recognition systems must function across heterogeneous sensing conditions, particularly visible (VIS) and infrared (IR) spectra, where sensing physics, appearance statistics, and noise characteristics differ substantially. Cross-spectral person identification, often referred to as Visible–Infrared Person Re-Identification (VI-ReID) has historically been studied using single infrared bands such as near-infrared (NIR) or long-wave infrared (LWIR), frequently relying on image translation or modality synthesis between VIS and IR domains. However, such approaches do not capture the full complexity of multi-domain infrared sensing. The IARPA Janus Benchmark Multi-Domain Face (IJB-MDF) dataset introduces a cross-spectral recognition protocol requiring identity matching between VIS imagery and multiple infrared domains, including short-wave infrared (SWIR), medium-wave infrared (MWIR), and long-wave infrared (LWIR). Recent work on multi-domain biometric recognition using body embeddings demonstrated that body-based representations can outperform face-based embeddings for cross-spectral identification in MWIR and LWIR conditions, and established the first benchmark results using the SemReID architecture within the MDF protocol. This study further provided insights into inter-domain infrared relationships, transferability of VIS-pretrained models, the importance of local semantic body features, and effective training strategies for limited multi-domain data [ 37 ] . Motivated by these findings, the present work introduces the first mutual-information–based expressivity analysis of body embeddings across spectral domains under the MDF recognition protocol. The study centers on the SemReID foundation model [ 23 ] . This choice is supported by two complementary observations: prior comparative evaluation established SemReID as providing strong rank-accuracy performance and stable representation quality among contemporary transformer-based ReID architectures [ 40 , 23 ] , while subsequent work on multi-domain biometric recognition using body embeddings demonstrated its effectiveness for cross-spectral identity matching within the MDF framework [ 37 ] and on the LLCM benchmark [ 57 ] . Building on this foundation, we investigate whether the attribute-encoding hierarchies observed in visible-spectrum person re-identification remain consistent, shift, or intensify across SWIR, MWIR, and LWIR domains, providing new insight into the robustness, fairness, and deployability of biometric recognition under real-world sensing conditions. The principal contributions of this work are summarized as follows: • We introduce the first MI-based framework for analyzing body-attribute encoding in transformer-based person re-identification representations, enabling layer-wise characterization of semantic attribute dependence within a large-scale self-supervised Vision Transformer (ViT) foundation model, SemReID [ 23 ] . • Using Mutual Information Neural Estimation (MINE) [ 3 ] , we quantify and rank attribute encoding strength in deep feature spaces, revealing stable hierarchical structure in how body-related attributes are represented across network depth and spectral sensing conditions. • We extend visible-spectrum analysis on the BRIAR benchmark [ 11 ] to the first cross-spectral expressivity study under the MDF recognition protocol [ 25 , 36 , 37 ] , providing new insight into how feature–attribute dependencies evolve across visible, SWIR, MWIR, and LWIR domains relevant to real-world biometric deployment. II Related Works II-A Bias

Read the complete text