Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws
Xiyuan Wei, Ming Lin, Fanjiang Ye, Fengguang Song, Liangliang Cao, My T. Thai, Tianbao Yang · 2025
Read Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws by Xiyuan Wei, Ming Lin, Fanjiang Ye, Fengguang Song, Liangliang Cao, My T. Thai, Tianbao Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06699.txt.
Introduction
Xiyuan Wei and co-authors study model steering in a 2025 machine learning paper. In model steering, a trained model guides the training of a new model by picking or weighting its data. The trick is used in practice, but its basic logic has been poorly understood.
The authors build a theory for the approach, rooted in methods that guard against shifts in data. Their analysis shows why a good guide model can improve accuracy and data efficiency. They then design a new method for training image-text models with a guide, called DRRho-CLIP. Tests confirmed the theory, showed a better scaling pattern than training without a guide, and beat older rule-of-thumb methods.
Catalog record
| CREATOR | Xiyuan Wei, Ming Lin, Fanjiang Ye, Fengguang Song, Liangliang Cao, My T. Thai, Tianbao Yang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-06699
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws Xiyuan Wei, Ming Lin, Fanjiang Ye, Fengguang Song, Liangliang Cao, My T. Thai, Tianbao Yang Abstract This paper formalizes an emerging learning paradigm that uses a trained model as a reference to guide and enhance the training of a target model through strategic data selection or weighting, named model steering . While ad-hoc methods have been used in various contexts, including the training of large foundation models, its underlying principles remain insufficiently understood, leading to sub-optimal performance. In this work, we propose a theory-driven framework for model steering called DRRho risk minimization , which is rooted in Distributionally Robust Optimization (DRO). Through a generalization analysis, we provide theoretical insights into why this approach improves generalization and data efficiency compared to training without a reference model. To the best of our knowledge, this is the first time such theoretical insights are provided for the new learning paradigm, which significantly enhance our understanding and practice of model steering. Building on these insights and the connection between contrastive learning and DRO, we introduce a novel method for Contrastive Language-Image Pretraining (CLIP) with a reference model, termed DRRho-CLIP. Extensive experiments validate the theoretical insights, reveal a superior scaling law compared to CLIP without a reference model, and demonstrate its strength over existing heuristic approaches. Code is released at github.com/Optimization-AI/DRRho-CLIP 1 Introduction With the success of large foundation models, numerous companies and research groups have entered the race to develop state-of-the-art models. While the data and code are often proprietary, the resulting models are sometimes released publicly, such as the CLIP models from OpenAI ( Radford et al., 2021 ) and the Llama models from Meta ( Dubey et al., 2024 ) . This raises an intriguing question: “ How can we leverage public models to improve training of a target model on custom datasets? ” To study this question, we explore an emerging learning paradigm that leverages a trained model as a reference to guide and enhance the training through strategic data selection or weighting. We refer to this paradigm as model steering . Unlike the widely adopted knowledge distillation framework, model steering does not assume that the reference model is a stronger teacher; in fact, it can lead to the training of a model that ultimately surpasses the reference model in performance, i.e., enabling weak to strong generalization (cf. Figure 2 ). Figure 1 : Comparison between a target model (ViT-B/16) trained by the proposed DRRho-CLIP and the reference model it leverages. OpenAI CLIP (ViT-B/32) was trained on a private 400M dataset with 12.8B samples seen and 32768 batch size. DRRho-CLIP model was trained on DFN-192M with 1.28B samples seen and 5120 batch size, and using OpenAI CLIP as a reference model 2 2 2 Our training took 376 GPU hours on 8 H100 (2 days), OpenAI CLIP (ViT-L/14) model was trained on 256 V100 with 12 days, which gives an estimate of 256*12*24/11.6=6356 GPU hours for training ViT-B/32 as its FLOPs is 11.6 smaller. . A few works have studied learning with a reference model (LEAR) in different contexts and demonstrated its effectiveness in accelerating the training of various models, including classification models ( Mindermann et al., 2022 ; Evans et al., 2025 ) , large language models ( Lin et al., 2024 ; Xie et al., 2023 ) , CLIP models ( Evans et al., 2024a ; Evans et al., 2024b ) . An interesting idea emerging from these studies is to perform online data selection, sampling, or data weighting using the reference model through a shifted loss called the RHO loss \ell(\bm{\theta},\bm{z})-\ell(\bm{\theta}_{\text{ref}},\bm{z}) , where \theta denotes the target model to be learned, and \bm{\theta}_{\text{ref}} denotes the reference model, \bm{z} denotes a data sample, and \ell(\cdot,\cdot) denotes a loss of interest. This approach is intuitive in the sense that a data \bm{z} with a high loss of the current model \ell(\bm{\theta}_{t},\bm{z}) and a low loss of the reference model \ell(\bm{\theta}_{\text{ref}},\bm{z}) means that it has high learnability and should be used for training ( Mindermann et al., 2022 ) . However, the theory behind this approach is quite limited, especially on improving generalization, which hinders our understanding of its effectiveness and derivation of best practice for model steering. Mindermann et al. (2022) tried to motivate this approach through maximizing the likelihood of true labels of classification for a hold-out dataset based on the selection of informative data in the current mini-batch. Through Bayes’ theorem and the approximation of negative log-likelihoods by losses, their analysis yields a data selection strategy by selecting data in the mini-batch that have the top values of \ell(\bm{\theta}_{t},\bm{z})-\ell(\bm{\theta}_{\text{ref}},\bm{z}) , which is termed RHO loss selection. While their analysis offers some intuition into why the RHO loss is a reasonable choice for data selection, it falls short of providing guarantees on how and why this approach improves generalization with reduced training data. Moreover, there is no theoretical framework to guide the optimal selection or weighting of data to accelerate training effectively. The heuristic approach that selects data with top RHO losses in the mini-batch is not necessarily the best. To address this gap, this paper introduces a novel learning framework for enhancing the understanding and practice of model steering based on the RHO loss \ell(\bm{\theta},\bm{z})-\ell(\bm{\theta}_{\text{ref}},\bm{z}) . Here, we use the term RHO loss in a broader sense, which is not necessarily restricted to classification as derived in Mindermann et al. (2022) . Our framework builds on Distributionally Robust Optimization (DRO) . Traditional DRO seeks to minimize the worst-case risk over perturbed data distributions within an uncertainty set derived from the empirical distribution. We extend this idea by applying DRO to the RHO loss \ell(\bm{\theta},\bm{z})-\ell(\bm{\theta}_{\text{ref}},\bm{z}) across the training data, yielding a risk function for model steering, which we term DRRho risk. Leveraging the generalization theory of DRO, we derive theoretical generalization bounds of DRRho risk minimization, offering insights into how it enhances generalization. To the best of our knowledge, this work presents the first generalization theory for model steering, significantly advancing our understanding of its effectiveness. Our framework not only provides a theoretical foundation for existing heuristic approaches to data selection, sampling, or weighting but also offers improved practices for model steering. To illustrate its practical utility, we focus on contrastive language-image pretraining (CLIP) with a reference model. By leveraging the connection between contrastive loss and DRO ( Qiu et al., 2023 ) , we introduce a novel method for CLIP with a reference model, termed DRRho-CLIP, which utilizes the DRRho risk for each anchor data point. Our experiments demonstrate the effectiveness of DRRho-CLIP. For example, when using OpenAI’s CLIP (ViT-B/32) model as a reference, DRRho-CLIP with ViT-B/16 as the backbone trained from scratch on the DFN-192M dataset ( Fang et al., 2024 ) achieves 68.84% zero-shot classification accuracy on ImageNet-1K data. It outperforms OpenAI’s CLIP performance of 63.32% trained on a different 400M dataset, and OpenCLIP’s performance of 67.8% trained on the same data (cf. Footnote 2 ). In addition, our extensive experiments using various reference models and large-scale datasets reveal the following: (1) DRRho-CLIP achieves comparable performance to the standard CLIP training method (without a reference model) while using significantly less training data (e.g., a