Project-Probe-Aggregate: Efficient Fine-Tuning for Group Robustness
Beier Zhu, Jiequan Cui, Hanwang Zhang, Chi Zhang · 2025
Read Project-Probe-Aggregate: Efficient Fine-Tuning for Group Robustness by Beier Zhu, Jiequan Cui, Hanwang Zhang, Chi Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-09487.txt.
Introduction
Image-text models can learn shortcuts, leaning on background clues that happen to match the label instead of the real subject. Beier Zhu, Jiequan Cui, Hanwang Zhang, and Chi Zhang present a three-step fix in this 2025 paper, named Project-Probe-Aggregate. The goal is fair accuracy across hidden groups in the data, without anyone labeling those groups by hand.
First, image features are projected to strip out the label signal, training a deliberately biased classifier that exposes the shortcut. Its mistakes flag the minority cases that the shortcut fails. Group labels are then inferred, with a correction step, and group weights are pooled per class to build the final, debiased classifier. Theory in the paper says the scheme sharpens minority detection and is optimal for balanced group error. Tests show top worst-group accuracy with almost no tunable parts.
Catalog record
| CREATOR | Beier Zhu, Jiequan Cui, Hanwang Zhang, Chi Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-09487
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Project-Probe-Aggregate: Efficient Fine-Tuning for Group Robustness Beier Zhu, Jiequan Cui, Hanwang Zhang, Chi Zhang Abstract While image-text foundation models have succeeded across diverse downstream tasks, they still face challenges in the presence of spurious correlations between the input and label. To address this issue, we propose a simple three-step approach–Project-Probe-Aggregate ( PPA )–that enables parameter-efficient fine-tuning for foundation models without relying on group annotations. Building upon the failure-based debiasing scheme, our method, PPA , improves its two key components: minority samples identification and the robust training algorithm. Specifically, we first train biased classifiers by projecting image features onto the nullspace of class proxies from text encoders. Next, we infer group labels using the biased classifier and probe group targets with prior correction. Finally, we aggregate group weights of each class to produce the debiased classifier. Our theoretical analysis shows that our PPA enhances minority group identification and is Bayes optimal for minimizing the balanced group error, mitigating spurious correlations. Extensive experimental results confirm the effectiveness of our PPA : it outperforms the state-of-the-art by an average worst-group accuracy while requiring less than 0.01 % 0.01\% tunable parameters without training group labels. † † footnotetext: This work was done during Beier Zhu’s visit at AGI Lab, Westlake University. † denotes corresponding author. 1 Introduction Image-text foundation models Radford et al. 2021 ; Jia et al. 2021 ; Wang et al. 2022 —large models pre-trained on web-scale data—are becoming increasingly prevalent in real-world deployments. However, adapting these models to downstream tasks, whether through zero-shot transfer or fine-tuning, remains challenging due to fundamental issues of group robustness: they achieve low average test error but incur high risk on certain groups of examples Chuang et al. 2023 ; Zhang and Ré 2022 ; You et al. 2024 ; Yang et al. 2023a . This issue becomes particularly pronounced in the presence of spurious correlations Geirhos et al. 2020 , where a classifier relies on non-causal relationships that happen to align with class labels in training but break down during deployment. For example, consider the case where “waterbirds” predominantly appear in images with a “water” background. When trained on such a dataset, neural networks often rely on the background to recognize the objects. As a result, several fine-tuning methods have emerged to improve group robustness in foundation models Chuang et al. 2023 ; Zhang and Ré 2022 ; You et al. 2024 ; Yang et al. 2023a ; Phan et al. 2024b . Unlike conventional robust training frameworks, which typically re-train the entire model Liu et al. 2021 ; Liu et al. 2023 ; Nam et al. 2020 ; Nam et al. 2022 (which is computationally expensive) and/or require group labels Sagawa et al. 2019 ; Arjovsky et al. 2019 (which demands additional annotation costs), these fine-tuning methods prioritize parameter and annotation efficiency. They address spurious correlations by optimizing only a small subset of parameters without relying on training group labels. For instance, Zhang and Ré Zhang and Ré 2022 fine-tune a lightweight adapter and You et al . You et al. 2024 propose to optimize projection layers to improve distributional robustness. In this paper, we build on these ideas by focusing on efficient fine-tuning to enhance group robustness, i.e . , training linear probes. Method Worst-group Recall (%) Worst-group Precision (%) CA Zhang and Ré 2022 46.4 15.6 JTT Liu et al. 2021 78.6 24.1 PPA (ours) 80.4 44.3 Table 1: The precision and recall of the worst-group examples ( i.e . , the group with lowest validation accuracy) identified by biased models on the Waterbird training set Sagawa et al. 2019 using CLIP ResNet-50. Our PPA is more susceptible to suprious features, achieving higher worst-group recall and precision compared to previous methods. At a high level, in the absence of group annotations, a common strategy Li and Vasconcelos 2019 ; Nam et al. 2020 ; Liu et al. 2023 ; Liu et al. 2021 ; Zhang and Ré 2022 ; Zhang et al. 2022 ; Dagaev et al. 2023 involves first identifying minority groups by training a biased model that easily overfits to spurious features. Subsequently, a debiased model is trained using the inferred group labels. To improve the identification of minority groups, prior work proposes various strategies: Nam et al. 2020 ; Liu et al. 2023 employ generalized cross-entropy loss to amplify the model bias, JTT Liu et al. 2021 simply trains a model with empirical risk minimization (ERM) and tunes hyperparameters to exacerbate bias, CA Zhang and Ré 2022 leverages the correctness of zero-shot predictions and Dagaev et al. 2023 uses low-capacity networks to capture spurious features. In this paper, we leverage the pre-trained knowledge from the image-text foundation models to effectively induce bias, further enhancing the identification of minority groups. Specifically, we project out class proxies, provided by the text encoders, from the input image features and use only the remaining information to train the biased classifiers. Intuitively, removing class information forces the model to rely more heavily on spurious features for predictions. We provide theoretical analysis to support this intuition in Proposition 1 . Empirically, in Table 1 , we evaluate the quality of the inferred groups in the training datasets obtained by our method, and compare it to prior approaches Zhang and Ré 2022 ; Liu et al. 2021 . Specifically, we measure precision—the fraction of examples within the identified minor groups that belong to the worst-performing group, and recall—the fraction of examples from the worst-performing group that are captured by the identified minor groups. As expected, our method achieves higher precision and recall, demonstrating superior group identification. Given the inferred groups, typical debiased learning algorithms up-weight/up-sample minority group samples to enhance performance on these challenging instances Liu et al. 2021 ; Kirichenko et al. 2022 ; Idrissi et al. 2022 ; Nam et al. 2020 ; Dagaev et al. 2023 . However, determining the optimal weights requires careful hyper-parameter tuning and lacks theoretical guidance. Since the goal of debiased training is to minimize the Balanced Group Error (BGE) Liu et al. 2023 ; Tsirigotis et al. 2024 , we propose a methodology designed to achieve this objective directly. Rather than predicting class labels, our method is trained to probe for group targets by compensating group priors into the softmax cross-entropy during training. Afterward, we aggregate group weights of each class to produce the final debiased classifier. We dub our overall debiasing framework Project-Probe-Aggregate (PPA), and demonstrate that it is Bayes optimal for minimizing BGE in Proposition 2 . In summary, our contributions are: • We project out the class proxies offered by the text encoder and train the biased model on the residual image features, with theoretical analysis showing this approach increases reliance on spurious features, improving minority group identification. • We propose a novel debiasing method that is consistent for minimizing the balanced group error. The debiased classifier predicts group labels compensated by group priors and uses weight-space aggregation after fine-tuning to avoid group inference overhead. • We confirm the effectiveness of our proposal across five benchmarks with spurious correlations. With less than 0.01\% of the foundation model parameters, our approach PPA improves the worst group accuracy over the state-of-the-art without training group annotations. Beyond linear