Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond
Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, Xiangzheng Zhang · 2025
Read Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond by Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, Xiangzheng Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10460.txt.
Introduction
Liang Wen and a large team share an open recipe for training long reasoning models, in a paper from 2025. Top models of this kind often rest on closed data, which blocks others from copying the work.
The team used only public data and models, with tasks that grow harder step by step, then rounds of fine tuning and reward based training. Their mid size model beat a well known rival on math tasks. Small sets of hard cases lifted small models to top scores for their size. The team posted the models, data, and code for free use.
Catalog record
| CREATOR | Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, Xiangzheng Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-10460
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, Xiangzheng Zhang Abstract This paper introduces Light-R1, an open-source suite for training long reasoning models using reproducible and cost-effective methodology. Given the proprietary nature of data used in the DeepSeek-R1 series, we develop an alternative approach leveraging exclusively public data and models. Our curriculum training progressively increases data difficulty, combined with multi-staged post-training. Our Light-R1-32B model, trained from Qwen2.5-32B-Instruct, outperforms DeepSeek-R1-Distill-Qwen-32B in math reasoning. Experimental results show that this curriculum approach becomes more effective when distinct, diverse datasets are available for different training stages: fine-tuning DeepSeek-R1-Distilled models (pre-tuned by DeepSeek team on proprietary data) with 3,000 challenging examples from our curriculum dataset yielded state-of-the-art 7B and 14B models, while the 32B model, Light-R1-32B-DS performed comparably to QwQ-32B and DeepSeek-R1. Furthermore, we extend our work by applying GRPO on long reasoning models. Our final Light-R1-14B-DS achieves SOTA performance among 14B models in math, with AIME24 & 25 scores of 74.0 and 60.2 respectively, surpassing many 32B models and DeepSeek-R1-Distill-Llama-70B. Despite math-focused training, Light-R1-14B-DS demonstrates strong cross-domain generalization. Light-R1 represents a significant advancement in making sophisticated reasoning models more accessible and implementable in real-world applications. Our models, training data and code have been made available at https://github.com/Qihoo360/Light-R1 . Figure 1: Reproducible state-of-the-art long COT models (top) developed from scratch (=short-COT base), (bottom) derived from DeepSeek-R1-Distill models (=long-COT base), via curriculum learning strategy. 1 Introduction Since the release of DeepSeek-R1 ( DeepSeek-AI 2025 ) , long chain-of-thought ( OpenAI 2024 ; Wei et al. 2022 ; Kimi 2025 ; Lightman et al. 2023 ) reasoning has gained widespread popularity in both foundational AI models and various industrial AI applications. However, deploying full-capacity R1-level models (typically 70B+ parameters, DeepSeek-R1 with 671B parameters) incurs prohibitive computational costs ( DeepSeek-AI 2025 ; Qwen 2025 ) . The resource barrier of training and deploying the giant models makes them impractical for edge devices and real-time applications. This limitation has sparked growing interest in developing compact yet capable models under a few 10B parameters that can perform extended long COT - a critical requirement for mathematical problem solving, algorithmic planning, and scientific analysis. To address this challenge, we present our work on the Light-R1 series. As a foundation for our research, we first established robust and reproducible evaluation protocols that rigorously reproduce the evaluation results reported in DeepSeek-AI 2025 . Building upon this reliable framework, our research systematically addresses three fundamental challenges through innovative algorithmic and engineering advancements. The first challenge involves curating an efficient dataset for Post-Training, a critical factor for long-COT optimization ( Ye et al. 2025 ; Muennighoff et al. 2025 ; Li et al. 2025 ) . We collected diverse open-source reasoning data covering mathematical reasoning, logical deduction, and algorithmic problem-solving. After preprocessing to remove duplicates and standardize formatting, we implemented a two-stage difficulty filtering methodology using DeepScaleR-1.5B-Preview ( Luo et al. 2025b ) and DeepSeek-R1-Distill-Qwen-32B models to quantify difficulty based on pass rates. The second challenge then emerges as how to optimize the utilization of this dataset. While conventional approaches typically employ a single SFT stage ( DeepSeek-AI 2025 ; Xu et al. 2025 ; Labs 2025 ; Yu et al. 2024 ) , our preliminary experiments with our 32B model revealed significant limitations—approximately 20% of training data still exhibited pass rates below 50% across 10 runs, indicating insufficient knowledge assimilation from heterogeneous difficulty datasets. To address this, we implemented a multi-staged curriculum training strategy comprising two consecutive SFT stages with progressively increasing difficulty, followed by a DPO stage ( Rafailov et al. 2023 ) . Although recent work has explored different curriculum strategies for long-COT training Luo et al. 2025a ; Min et al. 2024 ; Xi et al. 2024 ; Yuan et al. 2025a , our approach demonstrates superior performance: our Light-R1-32B model, trained from Qwen2.5-32B-Instruct ( Qwen 2024 ) , outperforms DeepSeek-R1-Distill-Qwen-32B in mathematical reasoning. The third challenge arises from implementing the final component of Post-Training — Reinforcement Learning ( Shao et al. 2024 ; Wang et al. 2024 ; Ouyang et al. 2022 ; Schulman et al. 2017 ; Schulman et al. 2015 ) — to further enhance model performance. We are excited to report our successful reinforcement learning training of Light-R1-14B-DS. While recent research has shown success in training base models ( Zeng et al. 2025 ; Hu et al. 2025 ; Liu et al. 2025 ) , smaller models ( Zeng et al. 2025 ; Luo et al. 2025b ) , or larger models with intensive computational resources ( Qwen 2025 ) , our long-COT RL Post-Training represents the first demonstration of simultaneous increases in both response length and reward scores on long-COT 14B models without the initial length reduction typically observed. This breakthrough demonstrates that carefully designed curriculum strategies can overcome the previously documented scalability limitations of RL in smaller models Gao et al. 2023 . The key contributions of this work include: • A detailed, fully open-source Curriculum Post-Training approach to train long-COT models from scratch. The multi-stage curriculum training incrementally builds reasoning capacity through difficulty-progressive data exposure, requiring only $1000 training cost (6 hours on 12×H800 GPUs). This approach is validated on Qwen2.5-32B-Instruct and could be easily migrated to 7B and 14B models. • A well established SFT stage 2 dataset of 3k mostly math questions that could significantly improve not only SFT stage 1 but also all DeepSeek-R1-Distill models, resulting in our SOTA 7B model Light-R1-7B-DS. • First demonstration of RL effectiveness on 14B models for mathematical reasoning, achieving around 2% absolute improvement compared with before-RL, resulting in our SOTA 14B model Light-R1-14B-DS. Table 1: Reproduction of DeepSeek-AI 2025 and Qwen 2025 evaluation results on AIME24 ( MAA 2024 ) pass@1 averaged over 64 runs. Model Paper Ours DS-distill-32B 72.6 72.3 DS-distill-14B 69.7 69.3 DS-distill-7B 55.5 54.0 QWQ-32B 79.5 78.5 2 The Origin of Everything: Stable and Trustworthy Evaluation of Long-COT Models Following DeepSeek-AI 2025 , long-COT models are commonly deployed with sampling temperature 0.6. While long-COT models generally perform better with sampling than with greedy decoding, it brings more burden for model evaluation as multiple samples for each question may be required, contrary to previous viable approaches of greedy decoding for evaluation ( Song et al. 2024 ) . DeepSeek-AI 2025 generates 64 responses per query to estimate pass@1. We have verified this choice, witnessing large deviation of over 3 points using 16 responses or fewer across different runs of the same model. Such randomness is unacceptable to compare model performances. For stable and trustworthy evaluation, we adapted Luo et al. 2025b ’s evaluation code for all our evaluation runs. Our evaluation code and logs are all released. We can reproduce all DeepSeek-R1-Distill models’ and QwQ’s scores as reported in DeepSeek-AI 2025 ; Qwen 2025 as shown in Tab. 1 with 64 samples per query, with deviation around 1 point. 3 Light-R1-32B: Long-COT from Scratch with Curriculum SFT & DPO While numerous studies ( Ye et al. 2025 ; Muennighoff et al. 2025 ; OpenThoughts 2025 ; OpenR1 2025 ) have open-sourced efforts to replicate