Exploring Weak-to-Strong Generalization for CLIP-based Classification

Jinhao Li [email protected], Sarah M. Erfani [email protected], Lei Feng [email protected], James Bailey [email protected], Feng Liu [email protected] · 2025

Read Exploring Weak-to-Strong Generalization for CLIP-based Classification by Jinhao Li [email protected], Sarah M. Erfani [email protected], Lei Feng [email protected], James Bailey [email protected], Feng Liu [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-18396.txt.

Introduction

Jinhao Li, Sarah M. Erfani, Lei Feng, James Bailey, and Feng Liu test whether a weak model can train a strong one for image and text sorting. Human review cannot keep pace as models grow past human skill in some lanes. A weak teacher that grades a strong pupil could ease that load, a plan shown to work for text only models.

The team extend the plan to CLIP, a model that sorts images by text match. Their method learns class core points, sharp typical forms of each class, to lift the strong model's sort skill. Gains would show the weak to strong plan ports to mixed media. The work probes a key route to safe growth of strong models.

Catalog record

CREATORJinhao Li [email protected], Sarah M. Erfani [email protected], Lei Feng [email protected], James Bailey [email protected], Feng Liu [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-18396

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Exploring Weak-to-Strong Generalization for CLIP-based Classification Jinhao Li [email protected], Sarah M. Erfani [email protected], Lei Feng [email protected], James Bailey [email protected], Feng Liu [email protected] Abstract Aligning large-scale commercial models with user intent is crucial to preventing harmful outputs. Current methods rely on human supervision but become impractical as model complexity increases. When models surpass human knowledge, providing accurate feedback becomes challenging and inefficient. A novel solution proposed recently is using a weaker model to supervise a stronger model. This concept leverages the ability of weaker models to perform evaluations, thereby reducing the workload on human supervisors. Previous work has shown the effectiveness of weak-to-strong generalization in the context of language-only models. Extending this concept to vision-language models leverages these insights, adapting the proven benefits to a multi-modal context. In our study, we explore weak-to-strong generalization for CLIP-based classification. We propose a method, class prototype learning (CPL), which aims to enhance the classification capabilities of the CLIP model, by learning more representative prototypes for each category. Our findings indicate that, despite using a simple loss function under weak supervision, CPL yields robust improvements in targeted scenarios, particularly when pretraining is limited. Extensive experiments demonstrate that our approach is effective under these settings, achieving a 3.67% improvement over strong baseline methods. 1 Introduction Large language models (LLMs), such as GPT 4o ( 1 ) , Claude 3 ( 4 ) and Gemini 1.5 ( 26 ) , have made significant strides in enhancing performance across a spectrum of natural language processing tasks. However, despite their successes, ensuring that these models align with human expectations and intentions remains a formidable challenge ( 6 ) . Increasing the size of language models does not necessarily improve their ability to follow user intent, as they can still produce untruthful, toxic, or unhelpful outputs, indicating a lack of alignment with their users ( 23 ) . Alignment with user intent is crucial for deploying these models effectively in practice ( 5 ) . Traditional alignment techniques often rely heavily on human supervision ( 7 ; 27 ; 23 ; 10 ; 5 ) , requiring evaluators to provide feedback on model outputs. However, as the complexity and intricacy of model outputs increase, the feasibility and scalability of this approach diminishes ( 6 ) . As a result, there is a need to effectively align LLMs with human values without overly burdening human evaluators. A recent study ( 6 ) explored a novel approach known as weak-to-strong generalization to address the challenge of aligning strong models with human feedback. This strategy leverages a weaker model to supervise a more robust one, presenting a promising method to enhance model alignment. The study by 6 demonstrates the effectiveness of this weak-to-strong learning approach, where finetuning strong models with knowledge generated by their weaker counterparts consistently improves performance. For example, in natural language processing tasks, finetuning GPT-4 with supervision from a GPT-2-level model significantly enhances GPT-4’s performance. This approach highlights the viability of weak-to-strong learning as a solution for better model alignment, demonstrating that even weaker models can provide valuable guidance for improving stronger models. While the technique proves effective, applying it to Vision-Language Models (VLMs) is far from straightforward. VLMs face unique challenges in aligning complex multimodal tasks, making it essential to thoroughly explore the method’s applicability and limitations in this context. Unlike in natural language tasks, where text-based guidance can be more straightforward, VLMs must align both visual and textual information, making supervision from weaker models significantly more challenging. The complexity of managing two distinct modalities introduces difficulties in ensuring coherent feedback across image and text domains, necessitating a more nuanced approach when adapting weak-to-strong generalization to VLMs. Our goal is to rigorously investigate the weak-to-strong paradigm within VLMs, as this problem extends beyond a mere adaptation of previous work. In this study, we explore weak-to-strong generalization for CLIP-based classification, recognizing it as a crucial starting point in VLMs. Existing VLMs ( 25 ; 14 ) take classification as the basic task to evaluate the alignment of images and texts. Also in our task setting, classification makes it easier for us to design simulation experiments and establish a benchmark. In this context, we introduce a method called class prototype learning (CPL). CPL involves generating class prototypes that encapsulate the characteristics of each class using weak supervision. This method effectively mitigates the false signals typically generated by weak supervision, thereby showcasing superior performance. Moreover, when compared to conventional methods for adapting VLMs to downstream tasks, such as prompt tuning ( 33 ; 15 ) , our CPL approach proves to be more efficient. This efficiency arises from the fact that CPL eliminates the need to employ a text encoder during the fine-tuning phase. By streamlining the adaptation process, CPL offers a more resource-effective solution while maintaining high-performance levels. We conduct extensive experiments to evaluate the performance of the proposed method, CPL, using the DomainNet dataset ( 24 ) , which includes six diverse visual domains. The dataset is divided into training and test sets, and the experiment involves training weak models, generating weak supervision sets, fine-tuning strong models with weak supervision, and comparing the results to strong model training with ground truth labels. Various baselines are used for comparison. Our results show that the CPL achieves the highest average accuracy across all domains, significantly outperforming other methods. In particular, CPL shows substantial improvements for challenging domains like Infograph and handles domain-specific features effectively, despite CLIP’s lower zero-shot performance in domains like QuickDraw. This illustrates the robustness and effectiveness of CPL in weak-to-strong generalization scenarios. We summarize the main contributions of our work: (i) Exploring weak-to-strong generalization for CLIP-based classification : Previous work ( 6 ; 11 ) has shown the effectiveness of weak-to-strong generalization in LLMs. Extending this concept to VLMs leverages these insights, adapting the proven benefits to a multi-modal context. (ii) Proposing CPL : We present a method that effectively leverages class prototype representations through weak supervision to enhance the classification performance of VLMs, such as CLIP ( 25 ) . (iii) Conducting simulation experiments : We design a simulation experiment within the VLMs framework based on DomainNet ( 24 ) to study this problem, and establish a benchmark in this context. Our experiment resulted in a 3.67% improvement over baseline methods. 2 Related Work Vision-language models. VLMs integrate visual and textual information, enabling a multifaceted understanding and interaction with multimodal content. CLIP ( 25 ) exemplifies this approach, leveraging contrastive learning to align images with textual descriptions effectively. This model demonstrates robust zero-shot capabilities, where it can recognize images or concepts it was not explicitly trained on. The effectiveness of CLIP and similar models, such as ALIGN ( 14 ) , Flamingo ( 3 ) , BLIP ( 19 ) and Llava ( 20 ) , arises from their ability to generalize from vast amounts of web-collected data, learning nuanced, multimodal representations that are applicable across various tasks and domains. Vision-language prompt tuning. Research has also focused on improving prompt-based learning and fine-tuning methods, such as CoOp ( 33 ) and CoCoOp ( 32 ) , which adapt VLMs more effectively to specific tasks by learning customized prompt strategies. CoOp transforms static text prompts into dynamic, learnable components.

Read the complete text