R ecall- E xtend D ynamics: Enhancing Small Language Models through Controlled Exploration and Refined Offline Integration
Zhong Guan, Likang Wu, Hongke Zhao, Jiahui Wang, Le Wu · 2025
Read R ecall- E xtend D ynamics: Enhancing Small Language Models through Controlled Exploration and Refined Offline Integration by Zhong Guan, Likang Wu, Hongke Zhao, Jiahui Wang, Le Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-16677.txt.
Introduction
Zhong Guan and co-authors present a training method called Recall-Extend Dynamics, or RED, for small language models, in a paper from 2025. Large models have gained skill at reasoning through reward-based training, but small models have seen less of that work. The team asks how to mix data drawn from larger models with training on the small model itself.
RED watches how the model's spread of choices shifts for each kind of data, and it adjusts the weight given to the taught data as training goes on. The method also deals with gaps between the taught data and what the model would now produce. Readers get a plan for giving small models more room to explore while keeping the taught data useful.
Catalog record
| CREATOR | Zhong Guan, Likang Wu, Hongke Zhao, Jiahui Wang, Le Wu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-16677
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
R ecall- E xtend D ynamics: Enhancing Small Language Models through Controlled Exploration and Refined Offline Integration Zhong Guan, Likang Wu, Hongke Zhao, Jiahui Wang, Le Wu Abstract Many existing studies have achieved significant improvements in the reasoning capabilities of large language models (LLMs) through reinforcement learning with verifiable rewards (RLVR), while the enhancement of reasoning abilities in small language models (SLMs) has not yet been sufficiently explored. Combining distilled data from larger models with RLVR on small models themselves is a natural approach, but it still faces various challenges and issues. Therefore, we propose R ecall- E xtend D ynamics(RED): Enhancing Small Language Models through Controlled Exploration and Refined Offline Integration. In this paper, we explore the perspective of varying exploration spaces, balancing offline distillation with online reinforcement learning. Simultaneously, we specifically design and optimize for the insertion problem within offline data. By monitoring the ratio of entropy changes in the model concerning offline and online data, we regulate the weight of offline-SFT, thereby addressing the issues of insufficient exploration space in small models and the redundancy and complexity during the distillation process. Furthermore, to tackle the distribution discrepancies between offline data and the current policy, we design a sample-accuracy-based policy shift mechanism that dynamically chooses between imitating offline distilled data and learning from its own policy. Code — https://github.com/millioniron/OpenRLHF-Millioniron-/tree/master Introduction Recent studies ( Jaech et al. 2024 ; Guo et al. 2025 ; Team et al. 2025 ) indicate that LLMs can achieve significant performance improvements in various tasks through enhanced reasoning capabilities. A key technology driving this progress is RLVR ( Lambert et al. 2024 ; Shao et al. 2024 ) , which has garnered widespread attention in the research community and led to the development of various policy optimization algorithms such as GRPO ( Guo et al. 2025 ) , DAPO ( Yu et al. 2025 ) , GPG ( Chu et al. 2025 ) , and DR.GRPO ( Liu et al. 2025 ) , and others ( Chen et al. 2025a ; Zhang et al. 2025a ) . These methods are characterized by their simplicity, directness, and result-oriented approach, distinguishing them from traditional methods based on process reward modeling ( Li and Li 2025 ; Setlur et al. 2024 ) . Despite the promising advancements in the aforementioned reasoning-related research, most studies have focused on models ranging from 7B to 32B in parameters. For smaller models (e.g., 1.5B), research has primarily adopted direct or multi-stage distillation methods ( Yang et al. 2025 ; Guo et al. 2025 ; Abdin et al. 2025 ) , and their integration with RLVR remains largely unexplored. However, small models trained solely through Supervised Fine-Tuning (SFT) distillation still exhibit noticeable issues such as overthinking and redundant generation ( Zhang et al. 2025c ; Li et al. 2025 ) , indicating room for overall performance improvement. As illustrated in the Figure 1 , pre-SFT models optimized with post-RL show improved reasoning performance and can, to some extent, reduce the probability of generating overthinking words, thereby decreasing redundant output. Figure 1: RL training on an SFT-distilled model. The pre-SFT model overthinks significantly, generating a lot of redundant content that leads to inefficient training. There’s still room to improve the model’s performance. Nevertheless, the “pre-SFT + post-RL” training pipeline still faces efficiency challenges. As depicted in the Figure 1 , the model’s generation length during RL training gradually decreases from an initial over 6,000 tokens to approximately 3,000, while the rollout time, influenced by generation length, significantly increases, further prolonging the training cycle. Therefore, this paper proposes a more efficient strategy combining offline-SFT with online-RL, aiming to enhance the reasoning capabilities of small-scale models. Compared to existing SFT + RL methods ( Yan et al. 2025 ; Fu et al. 2025 ; Zhang et al. 2025b ; Huang et al. 2025 ) , our innovation lies in exploring changes in the exploration space to balance offline distillation data with online reinforcement learning, alongside specifically designing and optimizing for the integration problem in offline data. (I) Controlled Exploration: Balancing Recall and Extend via Dynamic Entropy Regulation Existing research has confirmed that RLVR, in most cases, does not endow models with new capabilities but rather acts as an “amplifier”, uncovering and activating existing knowledge from the model’s pre-training phase ( Yue et al. 2025 ; Shao et al. 2024 ) . Specifically, RLVR demonstrates performance improvement for smaller k in pass@ k , but its effect declines for larger k . We define this process as a Recall phase, where the core objective is to optimize the reasoning path within existing capabilities while contracting the exploration space. SFT, conversely, can expand the reasoning boundary, introducing new reasoning patterns learned from stronger teacher models and increasing the model’s explorable space, thus seen as an Extend of the model’s capabilities ( Guo et al. 2025 ; Kim et al. 2025 ) . These two approaches can complement each other to some extent. We propose using the entropy variation ratio, an intuitive metric for explorable space, to balance the Recall and Extend phases, addressing the issues of insufficient RL exploration space in small models and the redundancy and complexity problems inherent in the distillation process, thereby achieving effective complementarity between the two. (II) Adaptive Integration of Offline Data with Accuracy-aware Policy Shifts Integrating distilled offline data into the policy optimization function can achieve a unified training paradigm. However, using a fixed clip value makes it difficult to handle distillation data that often deviates significantly from the current policy. Furthermore, if distillation samples are treated as a deterministic policy (i.e., setting \pi_{\text{offline}}=1 ), it can rapidly exacerbate entropy collapse; conversely, directly integrating distillation samples via SFT loss or an on-policy form may lead to model performance collapse. To address these issues, we propose a method that dynamically adjusts the policy offset based on the answer’s correctness rate. For samples with high correctness rates, we prefer setting \pi_{\text{offline}}=1 , allowing the model to learn from its own policy; for samples with low correctness rates, the policy offset is set closer to \pi , encouraging the model more inclined to imitate the distillation samples. This method not only improves the model’s adaptability and robustness to data of varying quality but also optimizes overall training efficiency. In brief, our contributions can be summarized as follows: • We frame the integration of RL and SFT as a synergy process of Recall and Extend. By dynamically balancing these two stages, we enable Extend phase to satisfy the exploration space required by Recall phase, while Recall refines and simplifies the complex information from Extend. This achieves steady improvement in smaller models using offline distilled data. • Our approach significantly improves integration of offline distilled data into the RL process. We introduced a policy offset term based on Accuracy-aware to estimate \pi_{\text{offline}} . This mechanism allows the model to adjust autonomously when processing samples with high accuracy, while leaning more towards imitating the large model’s strategy for samples with high error rates. This effectively avoids entropy collapse and performance degradation. Related Work Reasoning with Integrated SFT and RL UFT ( Wang et al. 2024 ) proposes a new method that combines the training paradigms of DPO