Reasoning Under 1 Billion: Memory-Augmented Reinforcement Learning for Large Language Models
Hung Le, Dai Do, Dung Nguyen, Svetha Venkatesh · 2025
Read Reasoning Under 1 Billion: Memory-Augmented Reinforcement Learning for Large Language Models by Hung Le, Dai Do, Dung Nguyen, Svetha Venkatesh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-02273.txt.
Introduction
Hung Le, Dai Do, Dung Nguyen, and Svetha Venkatesh taught small language models to reason better through memory-guided reinforcement learning in 2025. Learning by reward has sharpened big models' reasoning, but models under a billion parameters explore too weakly to gain much. Their weak start traps them in poor reasoning habits.
Episodic memory stores past reasoning chains that worked. An inner reward, computed by nearest-neighbor lookup in that memory, pays the model for finding fresh routes near proven ones. Controlled exploration balances novelty against reuse. On two math data sets, fine-tuning with this method lifted sample efficiency and generalization. The authors offer the route for reasoning gains in low-resource settings.
Catalog record
| CREATOR | Hung Le, Dai Do, Dung Nguyen, Svetha Venkatesh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-02273
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reasoning Under 1 Billion: Memory-Augmented Reinforcement Learning for Large Language Models Hung Le, Dai Do, Dung Nguyen, Svetha Venkatesh Abstract Recent advances in fine-tuning large language models (LLMs) with reinforcement learning (RL) have shown promising improvements in complex reasoning tasks, particularly when paired with chain-of-thought (CoT) prompting. However, these successes have been largely demonstrated on large-scale models with billions of parameters, where a strong pretraining foundation ensures effective initial exploration. In contrast, RL remains challenging for tiny LLMs with 1 billion parameters or fewer because they lack the necessary pretraining strength to explore effectively, often leading to suboptimal reasoning patterns. This work introduces a novel intrinsic motivation approach that leverages episodic memory to address this challenge, improving tiny LLMs in CoT reasoning tasks. Inspired by human memory-driven learning, our method leverages successful reasoning patterns stored in memory while allowing for controlled exploration to generate novel responses. Intrinsic rewards are computed efficiently using a kNN-based episodic memory, allowing the model to discover new reasoning strategies while quickly adapting to effective past solutions. Experiments on fine-tuning GSM8K and AI-MO datasets demonstrate that our approach significantly enhances smaller LLMs’ sample efficiency and generalization capability, making RL-based reasoning improvements more accessible in low-resource settings. 1 Introduction Large Language Models (LLMs) have demonstrated remarkable advancements in reasoning and problem-solving, driven by innovations in scaling strategies and training techniques Google (2024) ; OpenAI (2024a) . Despite its foundational role in defining LLM capability, scaling pre-training is prohibitively expensive and tends to plateau Xia et al. (2023) ; Hong et al. (2023) . As a result, post-training has become increasingly important, offering improvements in alignment, reasoning depth, and downstream task efficiency Kumar et al. (2025) . Among post-training approaches, reinforcement learning (RL) fine-tuning is a promising alternative to expensive LLM’s test-time search methods, such as MCTS or Beam Search Yao et al. (2023) ; Feng et al. (2023) ; Snell et al. (2024) . RL directly instills chain-of-thought (CoT) reasoning strategies into the model, enabling efficient deployment. Recent works like DeepSeek-R1 Guo et al. (2025) show that RL with simple outcome rewards can enhance reasoning without relying on heavy inference-time compute methods OpenAI (2024a) ; OpenAI (2024b) and complicated process-based rewards Lightman et al. (2023) ; Zhang et al. () . However, these benefits have been observed mainly in large models (8B–670B) Guo et al. (2025) ; Arora and Zanette (2025) ; Yeo et al. (2025) . In contrast, RL remains challenging for tiny LLMs, which we consider as having \leq 1B parameters . These weak models frequently produce incorrect outputs during training, failing to receive any outcome reward. For example, a 0.5B model may repeatedly generate improperly formatted answers to math questions, failing to produce any valid outputs that qualify for a reward. As a result, reward signals are extremely sparse . A common mitigation approach is using heuristic format-based rewards Guo et al. (2025) . However, we argue that relying on format-based rewards can cause training collapse in tiny LLMs, as they may overfit to simple format patterns while neglecting the main task. Worse, exploration is ineffective —not only because small models choose poor actions but also because they lack an explicit exploration mechanism. Unlike RL agents, LLMs do not actively explore or exploit; they passively sample from learned distributions. As noted in Krishnamurthy et al. (2024) , even large models struggle with exploration and exploitation; this issue becomes acute for small models. Finally, the lack of quality data in downstream tasks poses an additional challenge for training tiny LLMs with RL. Drawing inspiration from the human brain’s episodic memory, which stores and retrieves experiences to guide learning McClelland et al. (1995) , we introduce Memory-R + , a memory-augmented reinforcement learning framework designed to enhance CoT reasoning in tiny LLMs. To address the challenges of reward sparsity and insufficient exploration, we implement an intrinsic motivation mechanism that emulates the brain’s drive to seek successful outcomes (exploit) and avoid repeated errors (explore). This mechanism guides reasoning trajectories by leveraging two distinct episodic memory modules: one dedicated to storing successful reasoning traces and the other to capturing failed attempts. By employing nearest-neighbor estimation within a shared representation space, Memory-R + derives performance-driven intrinsic rewards from the memory. This process mirrors how humans learn from near-correct attempts, allowing LLMs to refine their reasoning by aligning with successful patterns while avoiding detrimental exploration paths. This intrinsic motivation effectively addresses the limitations of sparse external rewards, providing a continuous learning signal based on past experiences. Unlike traditional episodic control methods that rely on state-action-return associations for discrete action spaces Pritzel et al. (2017) ; Le et al. (2021) ; Le et al. (2022) ; Do et al. (2024) , Memory-R + simplifies memory storage to input-output pairs, making it more suitable for LLM’s textual reasoning. Upon receiving a new query, the framework retrieves outputs from similar past instances by first encoding the query and searching for the top-k most similar queries in memory using cosine similarity. The corresponding response sets from the success and failure memories are then retrieved. The exploitation reward is computed by measuring the Euclidean distance between the generated response and the centroid of the successful response set, encouraging the model to align with generalizable successful patterns rather than memorizing specific past responses. In contrast, the exploration reward is derived from the maximum cosine similarity between the generated response and the stored failure responses, ensuring that the model discovers novel outputs differing from incorrect reasoning. To maintain stability in training, both rewards are normalized within a sliding window, adapting to the model’s recent performance trends. To evaluate our approach, we conduct extensive experiments across several tiny LLMs on mathematical problem-solving. Our results demonstrate that Memory-R + significantly improves reasoning accuracy and robustness compared to baseline RL and other handcraft rewards. Moreover, analytic studies provide insights into training collapses and the impact of different memory configurations, highlighting the role of episodic memory in enhancing reasoning performance. In summary, our key contributions are as follows: (1) We pioneer an RL fine-tuning approach for tiny LLMs. (2) We introduce a memory-based intrinsic reward mechanism to teach LLMs to explore and exploit. (3) We empirically identify and analyze training collapse issues when fine-tuning tiny LLMs with RL. (4) Extensive experiments on CoT reasoning tasks show that Memory-R + outperforms other RL methods, enhancing reasoning in small models significantly. The significance of our approach is that our method enables effective RL fine-tuning for models as small as 500M parameters—orders of magnitude smaller than current state-of-the-art LLMs used in RL-based reasoning research Guo et al. (2025) . This dramatically lowers the barrier to entry for small research labs, academic groups, and companies with limited computing resources, making advanced reasoning capabilities more accessible. 2 Method 2.1 Intrinsic Reward Formulation for CoT Reasoning When the LLM generates a response to a given query, it receives two forms of feedback: an outcome reward R from an Answer Verifier that judges the correctness of the final answer extracted from the response, and an intrinsic reward R_{mem} from memory that reflects how the response aligns with past successes and failures. We note that the Answer Verifier can only assess the final answer and cannot evaluate the quality