DiffInf: Influence-Guided Diffusion for Supervision Alignment in Facial Attribute Learning
Basudha Pal, Rama Chellappa · 2026
Read DiffInf: Influence-Guided Diffusion for Supervision Alignment in Facial Attribute Learning by Basudha Pal, Rama Chellappa (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-06399.txt.
Introduction
Pal and his team set out DiffInf, a diffusion method guided by self influence that mitigates label inconsistencies. Facial attribute classification relies on labeled datasets where traits like age are unclear yet sorted into labels. A sorter is trained and self scores found to identify harmful instances. Instead of dropping influential samples, correction via a diffusion autoencoder aligns content with assigned labels while keeping identity. Edited samples take the place of the old ones, yielding a refined dataset. Across facial attribute work, DiffInf improves reach over noisy label training.
Catalog record
| CREATOR | Basudha Pal, Rama Chellappa |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34681 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.06399
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DiffInf: Influence-Guided Diffusion for Supervision Alignment in Facial Attribute Learning Basudha Pal, Rama Chellappa Abstract Facial attribute classification relies on large-scale annotated datasets in which many traits, such as age and expression, are inherently ambiguous and continuous but are discretized into categorical labels. Annotation inconsistencies arise from subjectivity and visual confounders such as pose, illumination, expression, and demographic variation, creating mismatch between images and assigned labels. These inconsistencies introduce supervision errors that impair representation learning and degrade downstream prediction. We introduce DiffInf, a self-influence–guided diffusion framework for mitigating annotation inconsistencies in facial attribute learning. We first train a baseline classifier and compute sample-wise self-influence scores using a practical first-order approximation to identify training instances that disproportionately destabilize optimization. Instead of discarding these influential samples, we apply targeted generative correction via a latent diffusion autoencoder to better align visual content with assigned labels while preserving identity and realism. To enable differentiable guidance during correction, we train a lightweight predictor of high-influence membership and use it as a surrogate influence regularizer. The edited samples replace the originals, yielding an influence-refined dataset of unchanged size. Across multi-class facial attribute classification, DiffInf consistently improves generalization compared with standard noisy-label training, robust optimization baselines, and influence-based filtering. Our results demonstrate that repairing influential annotation inconsistencies at the image level enhances downstream facial attribute classification without sacrificing distributional coverage. [ orcid=0009-0009-0920-8565] [orcid=0000-0002-7638-1650] keywords Guided Diffusion Models ,Influence Functions ,Noisy Labels ,Facial Attribute Learning ,Data-Centric Robustness † † credit: Conceptualization, Methodology, Software, Validation, Formal Analysis, Writing - Original Draft, Writing - Review & Editing † † credit: Supervision, Project administration, Investigation, Methodology, Resources, Writing - Review & Editing † † corresponding: Corresponding author 1 Introduction Deep neural networks underpin modern face analysis systems for facial attribute prediction, demographic inference, and biometric understanding across security, healthcare, and human–computer interaction. Their performance, however, is fundamentally constrained by the semantic consistency of supervisory signals. Many facial attributes, including age and other soft biometric traits, are inherently ambiguous and continuous yet discretized into multi-class categories. Their visual manifestation is influenced by confounders such as illumination, ethnicity, cosmetics, lifestyle, and pose, leading to annotation uncertainty and label noise in large-scale datasets 25 ; 38 ; 30 ; 15 . When image appearance and assigned labels diverge, models receive contradictory supervision, corrupting representation learning and degrading generalization, calibration, and fairness 37 ; 3 ; 19 . Data attribution and influence estimation offer tools to quantify how individual training samples shape model behavior. Classical influence functions 18 and scalable approximations 33 ; 29 estimate the contribution of each training instance to optimization dynamics and predictions. These approaches have been used to analyze memorization 6 , detect mislabeled or bias-inducing samples 25 , enable selective forgetting 8 , and improve robustness under noisy supervision 10 ; 4 ; 39 . A consistent finding is that a small subset of highly influential samples can disproportionately govern the learned decision surface, particularly under annotation ambiguity 6 ; 36 . Existing methods typically respond through removal, reweighting, or relabeling 14 ; 7 , leaving open the question of whether influential image–label inconsistencies can instead be addressed through direct generative refinement. Removal-based strategies can deplete the training distribution. When a high-influence sample is discarded due to supervision mismatch, the learner loses more than a corrupted label: it also loses a visually valid instance that may encode rare covariate combinations important for representing the full data manifold. By shifting from exclusion to semantic conservation, we treat such samples as informational assets requiring alignment rather than supervisory noise to be eliminated. This perspective preserves diversity and rare modes while stabilizing optimization through more consistent supervisory signals. In parallel, diffusion models have emerged as state-of-the-art generative frameworks 12 ; 34 ; 23 , demonstrating strong realism and controllability, including applications in bias mitigation 28 ; 27 , privacy preservation 21 , face swapping 17 , and related facial tasks 1 ; 13 ; 31 . Beyond unconditional synthesis, diffusion enables identity-preserving semantic editing via partial denoising 22 , cross-attention control 11 , CLIP-guided manipulation 16 , and instruction- or mask-based editing 2 ; 5 . These advances suggest the possibility of using generative models to reduce image–label inconsistency by aligning visual appearance with assigned attribute labels. A naive strategy for data refinement would be to use a pretrained facial-attribute classifier to guide the diffusion process. However, this creates a circular dependency. If the guidance model is trained on the same task as the downstream model, it inherits the same biases and errors 3 ; 40 that the refinement process is meant to correct, and may merely reinforce supervisory mistakes. Moreover, a static classifier does not identify which images most destabilize training for the learner of interest. This motivates a model-aware framework in which influence determines which samples to correct, and diffusion determines how to correct them. We introduce DiffInf, a self-influence–guided diffusion framework for facial attribute learning under noisy supervision. DiffInf is attribute-agnostic and applicable to multi-class attributes with ambiguous visual boundaries. We evaluate it on multi-class facial age prediction and facial expression recognition, building upon established formulations in the field 35 ; 9 ; 20 ; 24 . A baseline classifier is first trained, after which self-influence scores are computed using a first-order checkpoint-based approximation 33 to quantify each sample’s impact on training dynamics. High-influence samples with potential image–label inconsistencies are then targeted for correction. Instead of removing these samples, DiffInf performs generative supervision refinement with a pretrained latent diffusion autoencoder 32 . We optimize latent variables so that corrected samples become identity-preserving, visually realistic, and less training-disruptive under their assigned labels. The edited images replace the original influential samples, yielding an influence-refined dataset of unchanged size. Across tasks, influence-guided generative replacement consistently improves generalization, robustness, and calibration. DiffInf complements recent influence-aware robust learning approaches such as GRASP-PsONet 26 ; whereas such methods suppress the effects of noisy samples during optimization, DiffInf uses influence as a localization signal for direct data-level correction. The main contributions of this study are summarized as follows: • We introduce DiffInf, a self-influence–guided diffusion framework that aligns training images with assigned facial attribute labels under noisy supervision. • We propose targeted generative replacement of high-influence samples via latent diffusion autoencoding, producing an influence-refined dataset while preserving data coverage. • We incorporate a differentiable predictor of high-influence membership to enable scalable influence-guided correction during latent optimization. • We demonstrate consistent improvements in facial attribute prediction, particularly for multi-class age and expression classification, and show that repairing influential mismatches is more effective than simply removing them. Figure 1: The DiffInf framework for influence-guided generative supervision alignment. The process begins with a semantic encoder and conditional DDIM encoding to project the input image into a latent representation. During decoding, we perform iterative latent optimization guided by a composite loss function. This objective balances identity preservation (via a face recognition/verification network), influence suppression (via a dedicated predictor of high-influence membership), and semantic alignment while maintaining realism. By targeting high-influence samples for generative correction, DiffInf realigns training images with their assigned attribute labels while maintaining identity fidelity and structural realism. 2 Proposed Method Our approach, DiffInf (Fig. 1 ), addresses facial attribute learning under noisy supervision by explicitly repairing high-impact training