Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn’t
Quy-Anh Dang, Chris Ngo · 2025
Read Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn’t by Quy-Anh Dang, Chris Ngo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16219.txt.
Introduction
Quy-Anh Dang and Chris Ngo present a 2025 paper on teaching a small language model to reason with reinforcement learning on a shoestring. Big reasoning models usually need vast compute. Working with a one-and-a-half-billion-parameter model, four mid-range graphics cards, a day of training, and a compact set of seven thousand math problems, the team adapted a group-based policy method. Three experiments probed what happens as training proceeds.
Reasoning scores jumped fast: one contest benchmark rose from sixty-three to eighty percent, and another hit a mark above a noted commercial preview model. The whole run cost about forty-two dollars. Longer training, though, brought instability and length problems. The authors release their code and data openly. They argue reinforcement learning offers a cheap, honest route to reasoning in small models, with trade-offs now mapped.
Catalog record
| CREATOR | Quy-Anh Dang, Chris Ngo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-16219
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn’t Quy-Anh Dang, Chris Ngo Abstract Enhancing the reasoning capabilities of large language models (LLMs) typically relies on massive computational resources and extensive datasets, limiting accessibility for resource-constrained settings. Our study investigates the potential of reinforcement learning (RL) to improve reasoning in small LLMs, focusing on a 1.5-billion-parameter model, DeepSeek-R1-Distill-Qwen-1.5B , under strict constraints: training on 4 NVIDIA A40 GPUs (48 GB VRAM each) within 24 hours. Adapting the Group Relative Policy Optimization (GRPO) algorithm and curating a compact, high-quality mathematical reasoning dataset, we conducted three experiments to explore model behavior and performance. Our results demonstrate rapid reasoning gains - e.g., AMC23 accuracy rising from 63% to 80% and AIME24 reaching 46.7%, surpassing o1-preview - using only 7,000 samples and a $42 training cost, compared to thousands of dollars for baseline models. However, challenges such as optimization instability and length constraints emerged with prolonged training. These findings highlight the efficacy of RL-based fine-tuning for small LLMs, offering a cost-effective alternative to large-scale approaches. We release our code and datasets as open-source resources, providing insights into trade-offs and laying a foundation for scalable, reasoning-capable LLMs in resource-limited environments. All are available at https://github.com/knoveleng/open-rs . 1 VNU University of Science, Vietnam 2 Knovel Engineering Lab, Singapore {quyanh.dang, chris.ngo}@knoveleng.com 1 Introduction Recent advancements in large language models (LLMs) have significantly advanced the pursuit of artificial general intelligence (AGI), with models such as GPT-4o ( OpenAI 2024a ) , Claude 3.5 Sonnet ( Anthropic 2024 ) , and Gemini 1.5 ( Google 2024 ) demonstrating unprecedented capabilities. A pivotal aspect of this progress is the integration of post-training techniques into the training pipeline. These methods - including supervised fine-tuning (SFT) and reinforcement learning (RL) - enhance reasoning accuracy, align models with societal values, and adapt them to user preferences, all while demanding fewer computational resources than pre-training ( OpenAI 2024b ) . A notable innovation in this domain is OpenAI’s o1 series, which leverages inference-time scaling through extended Chain-of-Thought (CoT) reasoning to achieve remarkable performance in mathematics, coding, and scientific reasoning tasks ( OpenAI 2024b ) . However, despite these breakthroughs, scaling reasoning capabilities at test time remains a persistent challenge for the broader research community, largely due to limited access to proprietary methodologies and resources. Efforts to bolster LLM reasoning have explored diverse strategies. Process-based reward models ( Uesato et al. 2022 ; Lightman et al. 2023a ; Wang et al. 2023 ) guide models toward structured problem-solving, while RL approaches ( Kumar et al. 2024 ) optimize performance through feedback-driven learning. Search algorithms, such as Monte Carlo Tree Search (MCTS) and Beam Search, have also been employed to enhance reasoning depth ( Feng et al. 2024 ; Xin et al. 2024 ; Trinh et al. 2024 ) . Although these methods have driven incremental gains, they fall short of the general reasoning prowess exhibited by the o1 series. Recently, the DeepSeek-R1 model ( DeepSeek-AI 2025 ) has emerged as a competitive alternative, utilizing RL with the Group Relative Policy Optimization (GRPO) algorithm. Built on the 671-billion-parameter DeepSeek-V3, DeepSeek-R1 matches o1’s reasoning performance ( DeepSeek-AI 2025 ) . Yet, the sheer scale and computational demands of such models - often exceeding hundreds of billions of parameters - render them impractical for self-hosting by most organizations outside major technology firms, limiting their broader adoption. In contrast, small LLMs, typically ranging from 1 to 10 billion parameters, present a resource-efficient alternative with potential for widespread deployment. Previous studies have demonstrated the feasibility of enhancing small LLMs through RL-based fine-tuning inspired by DeepSeek-R1 ( Luo et al. 2025 ; Team 2025b ) . However, these efforts often rely on expansive datasets (hundreds of thousands to millions of samples) or incur significant computational costs, undermining their accessibility for resource-constrained settings. This tension motivates two central research questions: 1. How do small LLMs behave when fine-tuned under strict resource constraints, such as limited computational power and training time? 2. Can their reasoning performance be elevated using an RL-based approach akin to DeepSeek-R1’s methodology, and if so, how? These questions naturally extend to a practical inquiry: If viable, how should such an approach be implemented for small LLMs, and if not, what are the fundamental limitations? Addressing these, we investigate the reasoning capacity of a 1.5-billion-parameter model, DeepSeek-R1-Distill-Qwen-1.5B , under stringent constraints: training on a cluster of 4 NVIDIA A40 GPUs (48 GB VRAM each) within a 24-hour window. Our methodology adapts the GRPO-based RL framework from DeepSeek-R1, tailoring it to the resource-limited context of small LLMs. We assess performance on a suite of mathematical reasoning benchmarks, a domain requiring structured, logical problem-solving that serves as a robust testbed for reasoning ability. Figure 1: Comparison of zero-shot pass@1 performance versus model size (left) and computational cost (right). Our Open-RS ( red point ) achieves the highest AIME24 score (46.7%), outperforming o1-preview (44.6%) and other models ( green points ). Additionally, Open-RS models exhibit the lowest computational cost at approximately $42. Our study yields three primary contributions: 1. We systematically analyze the reasoning potential of small LLMs under specific computational constraints, providing a practical lens on their scalability and deployment feasibility. 2. We offer actionable insights into the efficacy and challenges of RL-based fine-tuning for small LLMs, bridging the gap between theoretical advancements and real-world applicability. 3. We release our source code and curated datasets as open-source resources, fostering reproducibility and encouraging further exploration by the research community. Our findings illuminate the promise of RL-based methods to enhance small LLMs’ reasoning capabilities, achieving competitive performance with minimal resources ( Figure 1 ). Simultaneously, they reveal critical challenges - such as data efficiency, optimization stability, and length constraints - that must be addressed to fully realize this potential. These insights lay the groundwork for developing lightweight, reasoning-capable LLMs suitable for resource-constrained environments, advancing the democratization of advanced AI technologies. The remainder of this paper is structured as follows: Section 2 details our methodology, including data curation, RL algorithm, and reward design; Section 3 presents three experiments, their results, and comparative analyses; and Section 4 summarizes key findings. Additional details, including related work, discussion, hyperparameter setups, and supplementary results, are provided in the Appendix. 2 Methodology In this section, we outline our approach to optimizing the reasoning capabilities of small large language models (LLMs) under computational constraints. Our methodology comprises two primary components: (1) the curation of a high-quality, mathematics-focused dataset, and (2) the application of a resource-efficient reinforcement learning (RL) algorithm. These components are designed to balance performance gains with practical limitations, such as reduced computational overhead and privacy considerations. High-Quality Dataset Curation To minimize training costs while maximizing reasoning performance, we curate a compact, high-quality dataset tailored to mathematical reasoning. This dataset is derived from two existing sources: the s1 dataset ( Muennighoff et al. 2025 ) and the DeepScaleR dataset ( DeepSeek-AI 2025 ) . By filtering and refining these datasets, we ensure that our training data is both relevant and challenging, enabling efficient learning for small LLMs. s1 Dataset The s1 dataset ( Muennighoff et al. 2025 ) is a general-purpose reasoning corpus comprising 59,029 questions sourced from diverse domains, including NuminaMATH ( LI et al. 2024