Metis-SPECS: Decoupling Learning via Self-distilled Preference-based Cold Start for VLMs

Kun Chen, Peng Shi, Haibo Qiu, Zhixiong Zeng, Siqi Yang, Wenji Mao, Lin Ma · 2025

Read Metis-SPECS: Decoupling Learning via Self-distilled Preference-based Cold Start for VLMs by Kun Chen, Peng Shi, Haibo Qiu, Zhixiong Zeng, Siqi Yang, Wenji Mao, Lin Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-25801.txt.

Introduction

Kun Chen led six co-authors presenting Metis-SPECS. Peng Shi, Haibo Qiu, and Zhixiong Zeng helped. Siqi Yang, Wenji Mao, and Lin Ma joined. Their 2025 paper decouples multimodal learning via cold start. Reinforcement learning with verifiable rewards drives multimodal models. Most methods begin with supervised fine-tuning cold start. That initializes policy before reinforcement learning begins. Supervised cold start intertwines thinking with output format.

Instruction-style overfitting may result from that intertwining. Direct preference tuning beats supervised methods in cold start. Their SPECS approach uses self-distilled preference data. No larger teachers or manual annotation are needed. Preference training learns transferable surface criteria, not content. Reinforcement learning then handles deep thinking. Consistent gains appear across benchmarks.

Catalog record

CREATORKun Chen, Peng Shi, Haibo Qiu, Zhixiong Zeng, Siqi Yang, Wenji Mao, Lin Ma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-25801

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Metis-SPECS: Decoupling Learning via Self-distilled Preference-based Cold Start for VLMs Kun Chen, Peng Shi, Haibo Qiu, Zhixiong Zeng, Siqi Yang, Wenji Mao, Lin Ma Abstract Reinforcement learning (RL) with verifiable rewards has recently catalyzed a wave of “MLLM-r1” approaches that bring RL to vision language models. Most representative paradigms begin with a cold start, typically employing supervised fine-tuning (SFT), to initialize the policy before RL. However, SFT-based cold start adopts the reasoning paradigm intertwined with task solution and output format, which may induce instruction-style overfitting, weakens out-of-distribution generalization, and ultimately affects downstream RL. We revisit the cold start along two views, its training method and data construction, and introduce the Generalization Factor (GF) coefficient to quantify the generalization capability under different methods. Our empirical study finds that preference–based training methods (e.g. DPO) generalizes better than SFT-based methods in cold start. Motivated by this, we propose SPECS —a S elf-distilled, P r e ference-based C old S tart framework that decouples multimodal learning: (1) generates introspective preference data pairs via self-distillation, avoiding reliance on larger teachers or manual annotation; (2) performs preference–based training to learn, focusing on shallow, transferable surface-form criteria (format, structure, style) rather than memorizing content; and (3) hands off to RL with verifiable rewards for deep reasoning results. Experimental results across multiple multimodal benchmarks show that our decoupling learning framework yields consistent performance gains over strong baselines, improving MEGA-Bench by 4.1% and MathVista by 12.2%. Additional experiments indicate that SPECS contributes to reducing in-distribution “stuckness,” improving exploration, stabilizing training, and raising the performance ceiling. 1 1 1 Project Page: https://github.com/Kwen-Chen/SPECS-VL † † footnotetext: ∗ Equal contribution. † Corresponding author(s). 1 Introduction Recently, inspired by the success of DeepSeek-R1 ( Guo et al. 2025 ) , in effectively enhancing the reasoning capabilities of large models through reinforcement learning (RL) with verifiable reward ( Lambert et al. 2024 ; Guo et al. 2025 ) , a growing body of work has begun to apply RL directly to vision language models (VLMs). This has led to a wave of exciting “MLLM-r1” research ( Meng et al. 2025 ; Shen et al. 2025 ; Peng et al. 2025 ; Zhou et al. 2025 ; Zhang et al. 2025a ; Wang et al. 2025b ; Wang et al. 2025a ; Zheng et al. 2025 ; Ma et al. 2025 ) , which leverage similar principles to advance multimodal reasoning. Previous research has indicated that prior to RL, employing a pre-training or warm-up phase (which is termed “cold start” ), can significantly improve the readability, stability, and even the final performance of RL training ( Guo et al. 2025 ) . Currently, the most commonly used cold start strategy is supervised fine-tuning (SFT) , where the model is first fine-tuned on a set of high-quality Chain-of-Thought reasoning data to provide a better initial policy for the subsequent RL phase ( Wei et al. 2025 ; Yang et al. 2025 ; Huang et al. 2025 ; Deng et al. 2025b ) . This strategy enables the model to be trained on complex reasoning data during the cold start phase, thereby acquiring reasoning ability. The common understanding behind SFT-based cold start is that reasoning abilities, reasoning format and other learning objectives can be jointly learned during the cold start phase. However, such an SFT-based joint learning paradigm may largely affect the model’s generalization capability ( Wu et al. 2025 ; Chu et al. 2025 ) , and consequently degrade subsequent RL ( Chen et al. 2025a ) . This raises an important research issue of quantifying and improving the model’s generalization capability during cold start and working in concert with subsequent RL. To address the above limitations, we consider an alternative learning paradigm, which separates the learning process into hierarchical stages based on the idea that cold start phase focused more on shallow, surface-form learning to avoid prematurely getting stuck in in-distribution problem solving, while subsequent RL focuses on the deep-level learning of a solution to boost the overall performance ( Bengio et al. 2009 ) . Thus, the intuition of our adopting decoupling learning for multimodal reasoning is that the selection of pre-training methods in cold start needs to better support the subsequent RL, both in terms of generalization and by having separate objectives to facilitate better final results. Another important issue is the generation of cold start data. Previously, the prohibitive cost of human annotation has motivated a growing body of research to explore the use of synthetic data. This often involves using a more capable large model as a “teacher” to distill data for a smaller “student” model. ( Zhang et al. 2025b ; Yao et al. 2024 ; Xu et al. 2024 ; Huang et al. 2025 ) . However, when the capability gap between the teacher model and the student model is too large, it can lead to a decline in model performance ( Zhang et al. 2023 ) . Alternatively, the DeepSeek-R1-Zero paradigm ( Guo et al. 2025 ) first directly applies reinforcement learning to the base model for obtaining R1-Zero and then generates cold start data by zero model itself. This paradigm has achieved very remarkable performance; yet it still has the limitation of reliance on the SFT cold start and the constraints between SFT and subsequent RL, thereby leaving room for further improvement. In this paper, to examine the suitable cold start training method, we propose the Generalization Factor (GF) coefficient in Section 2 to quantify the generalization capability of the model and conduct an empirical study to evaluate different training methods. We identify that Direct Preference Optimization (DPO) ( Rafailov et al. 2023 ) based on preference data is a cold start approach that enables the model to have better performance. On this basis, we present the S elf-distilled P r e ference-based C old- S tart (SPECS) framework in Section 3 . By decoupling the learning objectives during DPO to focus on output format, we create a pre-aligned model that serves as a superior starting point for the final RL fine-tuning. Our experiments show that this method leads to more stable, efficient training, and a higher performance ceiling compared to the advanced and strong baseline. The main contributions of this paper can be summarized as follows. 1. We present the SPECS framework, a three-stage cold start strategy. It generates preference data through self-distillation, uses DPO for cold start training, and separates training objectives so that the model first aligns with output formats, providing a stronger starting point for RL. 2. We propose Generalization Factor as a metric to evaluate a model’s generalization capability under different cold start training methods by comparing its performance on in-distribution and out-of-distribution tasks. 3. We reveal the importance of Decoupling Learning between the cold-start and RL phases. This separation improves exploration and reduces the risk of the model getting stuck on in-distribution solutions. 4. Our experimental results prove that a preference-based DPO cold start gives the model stronger generalization ability. In terms of the model’s final results, it achieves consistent performance gains across benchmarks, improving MEGA-Bench by 4.1% and MathVista by 12.2% over strong baselines, demonstrating the effectiveness of the SPECS. 2 Empirical Investigation 2.1 Evaluating Degree of Generalization To evaluate

Read the complete text