Where Did Your Model Learn That? Label-free Influence for Self-supervised Learning
Nidhin Harilal, Amit Kiran Rege, Reza Akbarian Bafghi, Maziar Raissi, Claire Monteleoni · 2024
Read Where Did Your Model Learn That? Label-free Influence for Self-supervised Learning by Nidhin Harilal, Amit Kiran Rege, Reza Akbarian Bafghi, Maziar Raissi, Claire Monteleoni (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17170.txt.
Introduction
Nidhin Harilal and co-workers define data influence for self-supervised learning, where no labels exist, in a 2024 paper. Such models learn from huge unlabeled sets, yet which training examples shape them is poorly understood. Classic influence tools need labels, so they cannot be used.
Their method, Influence-SSL, judges an example by how stable the model's representation stays under changes to that example. Theory and experiments support the definition. Self-supervised models turned out to react to influential data differently from supervised ones. The team demonstrates uses in spotting duplicates, finding outliers, and auditing fairness. Code was released publicly for other researchers.
Catalog record
| CREATOR | Nidhin Harilal, Amit Kiran Rege, Reza Akbarian Bafghi, Maziar Raissi, Claire Monteleoni |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-17170
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Where Did Your Model Learn That? Label-free Influence for Self-supervised Learning Nidhin Harilal, Amit Kiran Rege, Reza Akbarian Bafghi, Maziar Raissi, Claire Monteleoni Abstract Self-supervised learning (SSL) has revolutionized learning from large-scale unlabeled datasets, yet the intrinsic relationship between pretraining data and the learned representations remains poorly understood. Traditional supervised learning benefits from gradient-based data attribution tools like influence functions that measure the contribution of an individual data point to model predictions. However, existing definitions of influence rely on labels, making them unsuitable for SSL settings. We address this gap by introducing Influence-SSL, a novel and label-free approach for defining influence functions tailored to SSL. Our method harnesses the stability of learned representations against data augmentations to identify training examples that help explain model predictions. We provide both theoretical foundations and empirical evidence to show the utility of Influence-SSL in analyzing pre-trained SSL models. Our analysis reveals notable differences in how SSL models respond to influential data compared to supervised models. Finally, we validate the effectiveness of Influence-SSL through applications in duplicate detection, outlier identification and fairness analysis. Code is available at: https://github.com/cryptonymous9/Influence-SSL . 1 Introduction Figure 1 : Illustration of scenarios for varying influence levels for self-supervised learning: (1) Hard negative examples, which challenge the model’s decision boundary, should have high influence. (2) Atypical examples, representing rare or outlier data, should also exhibit high influence. (3) Near-duplicate examples should maintain low influence due to their redundant nature and low contribution to learning. Influence functions [ 25 ] have proven valuable in supervised learning for assessing the impact of individual training examples on model behavior, providing insights into data memorization and learning dynamics [ 12 , 2 , 17 , 38 ] . However, extending this concept to SSL presents unique challenges, as existing definitions rely heavily on labeled data and often require computationally expensive model retraining [ 12 , 18 ] . Unlike supervised learning, where labels provide a direct link between training examples and model behavior, SSL tasks often involve pretext objectives that obscure how specific data points influence model training and performance. This makes extending influence estimation to SSL particularly challenging. This demands the development of a new theoretical framework for influence estimation tailored to SSL, one that can account for the distinct characteristics of SSL learning objective. The importance of understanding data influence in SSL extends beyond theoretical interest; it has practical implications for data curation, robustness analysis, and debiasing [ 2 ] . By quantifying the influence of training instances, practitioners can identify harmful or overly influential examples, leading to more stable and interpretable learning processes. Yet, the absence of labels and the complexity of pretext tasks mean traditional influence estimation techniques, which depend on gradient-based methods or retraining schemes [ 25 ] , are not directly applicable [ 21 ] . We introduce Influence-SSL , a novel framework for analyzing how training examples shape self-supervised representations. Our approach reformulates influence functions for the SSL setting by leveraging the fundamental invariance-distinctiveness trade-off inherent in SSL objectives. Rather than relying on labels, we quantify influence through representation stability against data augmentations, providing a natural measure aligned with SSL training objectives. We provide some intuition on influential examples computed using Influence-SSL in Figure 1 . We provide both a theoretical angle and empirical validation for understanding SSL dynamics using influence analysis. We show that Influence-SSL holds several mathematical properties required from influence estimation in a simplified setting. Additionally, we establish the connection between influence scores, representation structure, and augmentation consistency. Comprehensive experiments across various SSL frameworks confirm that our influence measures offer consistent and interpretable insights into model behavior. These insights enable practical applications such as duplicate detection, outlier identification, and fairness analysis. Our results indicate that influence analysis can uncover significant patterns in how SSL methods process training data, with potential benefits for enhancing model robustness and transparency. 2 Background 2.1 Classical Influence Functions In this section, we briefly review influence functions [ 25 , 2 ] , which are a classical technique from robust statistics [ 20 ] that was recently introduced to deep learning by [ 25 ] to help analyze how individual training examples affect a model’s learned parameters and predictions. Consider a dataset S=\{z_{i}\}_{i=1}^{n} where each z_{i} may represent either a single sequence (for self-supervised learning) or an input-target pair z_{i}=(x_{i},y_{i}) (for supervised learning). The model parameters \theta\in\mathbb{R}^{d} are learned through empirical risk minimization of a loss function \mathcal{L} : \theta^{*}=\argmin_{\theta\in\mathbb{R}^{d}}\frac{1}{n}\sum_{i=1}^{n}\mathcal{L}(z_{i},\theta) (1) To understand the effect of removing a new training example z_{m} to the dataset, we can parameterize the training objective by the weight \epsilon\in\mathbb{R} given to this example. This yields the response function: \displaystyle\theta^{*}(\epsilon) \displaystyle=\argmin_{\theta\in\mathbb{R}^{d}}J(\theta,S_{-\epsilon}) \displaystyle=\argmin_{\theta\in\mathbb{R}^{d}}\frac{1}{n}\sum_{i=1}^{n}\mathcal{L}(z_{i},\theta)-\epsilon\mathcal{L}(z_{m},\theta) The influence of z_{m} on \theta^{*} is defined as the first-order Taylor approximation to the response function at \epsilon=0 . Under appropriate regularity conditions, the Implicit Function Theorem yields: \mathcal{I}_{\theta^{*}}(z_{m})=\frac{d\theta^{*}}{d\epsilon}\bigg|{\epsilon=0}=-H^{-1}\nabla_{\theta}\mathcal{L}(z_{m},\theta^{*}) (2) where H=\nabla^{2}_{\theta}J(\theta^{*},S) is the Hessian of the loss. This allows us to linearly approximate the change in parameters when adding z_{m} with weight \epsilon=\frac{1}{n} : \theta^{*}(\epsilon)-\theta^{*}\approx I_{\theta^{*}}(z_{m})\epsilon=-H^{-1}\nabla_{\theta}\mathcal{L}(z_{m},\theta^{*})\epsilon (3) Since the direct parameter influence \mathcal{I}_{\theta^{*}} can be difficult to interpret, it is common to instead compute the influence on a measurable quantity g(\theta) , such as the validation loss or model predictions for a query point z_{q} . By the chain rule: \displaystyle\mathcal{I}_{f}(z_{m}) \displaystyle=\nabla_{\theta}g(\theta^{*})^{\top}\mathcal{I}_{\theta^{*}}(z_{m}) \displaystyle=-\nabla_{\theta}g(\theta^{*})^{\top}H^{-1}\nabla_{\theta}\mathcal{L}(z_{m},\theta^{*}) (4) Thus, one can efficiently analyze how an individual training examples affects both model parameters and downstream quantities of interest, without requiring explicit model retraining. In this paper, we are interested in the case where g=\mathcal{L}(z_{m},\theta^{*}) i.e. the influence of a point on itself, also referred to sometimes as self-influence [ 13 ] . 2.2 Self Supervised Learning In SSL, we train a model using only input data without explicit labels. Let x\in\mathcal{X} be an input, and \tau:\mathcal{X}\rightarrow\mathcal{X}\times\mathcal{X} be a stochastic transformation function that generates a pair of views (x_{a},x_{b})=\tau(x) . The model consists of an encoder f_{\theta}:\mathcal{X}\rightarrow\mathbb{R}^{d} that maps inputs to embeddings, where \theta\in\mathbb{R}^{D} represents the model parameters. Given a dataset S=\{x_{i}\}_{i=1}^{n} , the model is trained to maximize the similarity between embeddings of different views of the same input while minimizing similarity between embeddings of different inputs. The optimal parameters are found through empirical risk minimization: \theta^{*}=\argmin_{\theta\in\mathbb{R}^{D}}\frac{1}{n}\sum_{i=1}^{n}\mathcal{L}(x_{i},\theta) (5) Our setup enables the analysis of influence functions in the self-supervised setting by treating each x_{i} as a training example z_{i} , without requiring modification to the influence computation methods described above. 2.3 Related Work Data Attribution (DA) techniques explain model predictions by analyzing the training data that shaped the model [ 19 ] . DA methods are broadly categorized into retraining-based and gradient-based approaches. Retraining-based methods, including leave-one-out [ 11 , 13 ] , Shapley value techniques [ 31 , 14 ] , and Datamodels [ 23 ] , assess data impact by retraining the model on subsets, which is computationally expensive and limits scalability. Gradient-based methods estimate data influence via parameter sensitivity. Notable examples are representer point methods [ 33 ] , TracIn [ 29 ] , and influence functions [ 25 ] , the focus of this work. Extensions of influence functions explore group data effects [ 26 ] , higher-order information [ 4 ] , and normalized rankings [ 3 ] . The fundamental idea