TrustSkin: A Fairness Pipeline for Trustworthy Facial Affect Analysis Across Skin Tone
Ana M. Cabanas, Alma Pedro, Domingo Mery Departamento de Física, Universidad de Tarapacá, Arica, 1010069, Chile Departamento de Ciencia de la Computación, Facultad de Ingeniería, Pontificia Universidad Católica de Chile, Santiago, 7820436, Chile * Corresponding author: Ana M. Cabanas, acabanas@acade · 2025
Read TrustSkin: A Fairness Pipeline for Trustworthy Facial Affect Analysis Across Skin Tone by Ana M. Cabanas, Alma Pedro, Domingo Mery Departamento de Física, Universidad de Tarapacá, Arica, 1010069, Chile Departamento de Ciencia de la Computación, Facultad de Ingeniería, Pontificia Universidad Católica de Chile, Santiago, 7820436, Chile * Corresponding author: Ana M. Cabanas, acabanas@acade (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20637.txt.
Introduction
Ana Cabanas, Alma Pedro, and Domingo Mery audit face emotion software for bias across skin tone. Their 2025 study notes that fairness checks need a sound way to group faces by tone, and the common angle method bends under poor light. A second method, built on lightness and hue, is tested as a steadier yardstick.
Dark skin made up only about two percent of the test data, a sharp gap. Scores still split by group, with gaps in key fairness measures. The tone method built on lightness and hue grouped faces more evenly and gave clearer fault reads. Heat maps show the model looks at unlike face zones by tone. A modular fairness pipeline is proposed for future fixes.
Catalog record
| CREATOR | Ana M. Cabanas, Alma Pedro, Domingo Mery Departamento de Física, Universidad de Tarapacá, Arica, 1010069, Chile Departamento de Ciencia de la Computación, Facultad de Ingeniería, Pontificia Universidad Católica de Chile, Santiago, 7820436, Chile * Corresponding author: Ana M. Cabanas, acabanas@acade |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20637
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TrustSkin: A Fairness Pipeline for Trustworthy Facial Affect Analysis Across Skin Tone Ana M. Cabanas, Alma Pedro, Domingo Mery Departamento de Física, Universidad de Tarapacá, Arica, 1010069, Chile Departamento de Ciencia de la Computación, Facultad de Ingeniería, Pontificia Universidad Católica de Chile, Santiago, 7820436, Chile * Corresponding author: Ana M. Cabanas, [email protected] Abstract Understanding how facial affect analysis (FAA) systems perform across different demographic groups requires reliable measurement of sensitive attributes such as ancestry, often approximated by skin tone, which itself is highly influenced by lighting conditions. This study compares two objective skin tone classification methods: the widely used Individual Typology Angle (ITA) and a perceptually grounded alternative based on Lightness ( L ∗ L^{*} ) and Hue ( H ∗ H^{*} ). Using AffectNet and a MobileNet-based model, we assess fairness across skin tone groups defined by each method. Results reveal a severe underrepresentation of dark skin tones ( ∼ \sim 2 % \% ), alongside fairness disparities in F1-score (up to 0.08) and TPR (up to 0.11) across groups. While ITA shows limitations due to its sensitivity to lighting, the H ∗ H^{*} - L ∗ L^{*} method yields more consistent subgrouping and enables clearer diagnostics through metrics such as Equal Opportunity. Grad-CAM analysis further highlights differences in model attention patterns by skin tone, suggesting variation in feature encoding. To support future mitigation efforts, we also propose a modular fairness-aware pipeline that integrates perceptual skin tone estimation, model interpretability, and fairness evaluation. These findings emphasize the relevance of skin tone measurement choices in fairness assessment and suggest that ITA-based evaluations may overlook disparities affecting darker-skinned individuals. I INTRODUCTION Predictive algorithms and biometric systems are increasingly used in critical areas such as healthcare, security, and human-computer interaction [ 1 ] . However, these systems remain prone to bias arising from demographic imbalances in training data and algorithmic design flaws [ 2 , 3 , 1 ] . In computer vision applications like EmotionAI and Facial Affect Analysis (FAA), such biases often result in consistent performance disparities across attributes like age, sex, and skin tone [ 4 , 5 , 6 ] . Given the sensitive deployment of FAA in psychological evaluation, driver monitoring, and educational feedback [ 1 , 7 , 8 ] , ensuring fairness, transparency, and robustness across demographic groups is essential. A key challenge in fairness research is the reliable measurement of sensitive attributes such as ancestry or skin tone, which is highly affected by lighting conditions. The Gender Shades study exemplified this, showing error rates of 34.7 \% for darker-skinned women versus 0.8 \% for lighter-skinned men in commercial systems [ 9 ] . These disparities raise concerns about identity construction in facial datasets, where categories like gender and ancestry are often treated as fixed despite lacking standardized definitions [ 10 , 6 , 5 ] . Ancestry labels are socially constructed and inconsistently applied [ 11 ] , while skin tone annotations are influenced by lighting, sensor variation, and annotator bias [ 12 , 13 ] . Widely used scales such as the Fitzpatrick Skin Type (FST) [ 14 ] are biased toward lighter skin and unsuitable for fairness benchmarking [ 9 , 12 , 10 , 11 ] . As an alternative, the Individual Typology Angle (ITA) [ 15 , 16 , 17 ] is commonly used but presents critical limitations under real-world conditions [ 18 , 12 ] . Although skin tone bias has been extensively studied in face recognition systems [ 19 , 3 , 20 , 1 , 21 ] , it remains comparatively under-explored FAA [ 22 , 13 , 18 , 23 ] . While some fairness-aware strategies, such as objective tone classification and dataset rebalancing, have been proposed, their effectiveness in affective computing remains largely unvalidated [ 24 , 7 ] . Recent work has investigated fairness in FAA through diverse lenses, including algorithmic perception and trust [ 25 ] , counterfactual fairness [ 26 ] , multimodal bias analysis [ 27 ] , domain adaptation for demographic bias [ 28 ] , and evaluation frameworks that go beyond accuracy [ 29 ] . However, despite these advances, none of these studies explicitly address the role of skin tone as a source of bias in affective systems. Modeling skin tone under real-world conditions remains a persistent challenge, as illumination, sensor variation, and background context can introduce artifacts that distort both classification and subgroup attribution. Prior work in skin detection has shown that hybrid color space representations (e.g., RGB, HSV, YCbCr) offer more robust tone modeling than RGB alone [ 30 , 31 , 32 ] . This is particularly relevant to FAA, where widely used proxy metrics like the Individual Typology Angle (ITA) oversimplify skin tone and fail to account for lighting variability [ 33 ] . To address this, we propose a perceptually grounded measurement method based on Lightness ( L^{*} ) and Hue ( H^{*} ) to improve subgroup consistency and fairness evaluation under uncontrolled imaging conditions. We empirically assess this approach using the AffectNet dataset [ 34 ] and a MobileNet-based classifier, comparing fairness metrics across skin tone groups defined by both ITA and our L^{*} – H^{*} method. Our analysis reveals a severe demographic imbalance in the dataset, with less than 2% of samples representing the darkest skin tone group—echoing similar disparities observed in dermatology datasets [ 35 , 36 ] and computer vision benchmarks [ 9 , 12 ] . Furthermore, we show that ITA-based groupings produce erratic and potentially misleading fairness metrics for darker-skinned individuals, likely due to illumination-related measurement artifacts—a finding consistent with prior work [ 31 , 35 ] . In contrast, the L^{*} – H^{*} approach yields more stable subgroupings and fairness evaluations, though still limited by underrepresentation. Attempts to correct this imbalance through targeted data augmentation failed to improve fairness on the unbalanced test set, corroborating findings that post-hoc mitigation is ineffective when upstream bias persists [ 33 ] . II METHODOLOGY To investigate the impact of skin tone measurement on fairness in FAA, we compared two objective skin tone estimation methods. Our pipeline involved dataset selection, preprocessing, skin pixel segmentation, color space conversion, tone stratification using ITA and Hue-Lightness ( H^{*}L^{*} ), and subsequent demographic analysis. II-A Dataset and Preprocessing We used the AffectNet dataset [ 34 ] , a large-scale resource for facial affect analysis containing over one million facial images collected from the web. Approximately 450,000 of these images include manual annotations for categorical emotions (neutral, happy, sad, surprise, fear, disgust, anger, contempt). While AffectNet provides extensive variability in facial pose, lighting, and expression, it was not designed with demographic balance or fairness in mind, and includes a substantial number of grayscale or near-grayscale images. These characteristics limit its suitability for color-based skin tone analysis required in fairness evaluations. We applied a preprocessing pipeline to address these limitations to filter out grayscale and low-color content images from the training and validation splits. Face regions were automatically detected and cropped using MTCNN to minimize background interference and improve skin tone estimation. The images were then converted to the YCrCb color space. Skin pixels were segmented using a combination of Otsu’s thresholding and chrominance-based filtering, applied