ReMA: Learning to Meta-think for LLMs with Multi-agent Reinforcement Learning
Ziyu Wan, Yunxiang Li, Xiaoyu Wen, Yan Song, Hanjing Wang, Linyi Yang, Mark Schmidt, Jun Wang, Weinan Zhang, Shuyue Hu, Ying Wen · 2025
Read ReMA: Learning to Meta-think for LLMs with Multi-agent Reinforcement Learning by Ziyu Wan, Yunxiang Li, Xiaoyu Wen, Yan Song, Hanjing Wang, Linyi Yang, Mark Schmidt, Jun Wang, Weinan Zhang, Shuyue Hu, Ying Wen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-09501.txt.
Introduction
Ziyu Wan and a large team teach language models to think about their own thinking through two cooperating agents. One high agent plans and watches the work, while one low agent does the step by step solving. Both learn by trial reward with shared goals.
Split roles help the pair learn oversight skills that a lone agent learns poorly. Tests on hard math and judge style tasks beat lone agent training. A multi turn form adds tricks to stay fast. Studies of each agent show how the watch role grows and lifts the solver.
Catalog record
| CREATOR | Ziyu Wan, Yunxiang Li, Xiaoyu Wen, Yan Song, Hanjing Wang, Linyi Yang, Mark Schmidt, Jun Wang, Weinan Zhang, Shuyue Hu, Ying Wen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 89 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-09501
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ReMA: Learning to Meta-think for LLMs with Multi-agent Reinforcement Learning Ziyu Wan, Yunxiang Li, Xiaoyu Wen, Yan Song, Hanjing Wang, Linyi Yang, Mark Schmidt, Jun Wang, Weinan Zhang, Shuyue Hu, Ying Wen Abstract Recent research on Reasoning of Large Language Models (LLMs) has sought to further enhance their performance by integrating meta-thinking—enabling models to monitor, evaluate, and control their reasoning processes for more adaptive and effective problem-solving. However, current single-agent work lacks a specialized design for acquiring meta-thinking, resulting in low efficacy. To address this challenge, we introduce Re inforced M eta-thinking A gents (ReMA), a novel framework that leverages Multi-Agent Reinforcement Learning (MARL) to elicit meta-thinking behaviors, encouraging LLMs to think about thinking . ReMA decouples the reasoning process into two hierarchical agents: a high-level meta-thinking agent responsible for generating strategic oversight and plans, and a low-level reasoning agent for detailed executions. Through iterative reinforcement learning with aligned objectives, these agents explore and learn collaboration, leading to improved generalization and robustness. Empirical results from single-turn experiments demonstrate that ReMA outperforms single-agent RL baselines on complex reasoning tasks, including competitive-level mathematical benchmarks and LLM-as-a-Judge benchmarks. Additionally, we further extend ReMA to multi-turn interaction settings, leveraging turn-level ratio and parameter sharing to improve efficiency. Comprehensive ablation studies further illustrate the evolving dynamics of each distinct agent, providing valuable insights into how the meta-thinking reasoning process enhances the reasoning capabilities of LLMs. Our code can be found in https://github.com/ziyuwan/ReMA-public 1 Introduction Large language models (LLMs) have demonstrated remarkable capabilities in knowledge understanding and complex reasoning tasks ( Chowdhery et al. 2023 ; Achiam et al. 2023 ; Anil et al. 2023 ; Dubey et al. 2024 ) . The paradigm in developing LLM-based reasoning models is shifting from scaling training-time computation towards scaling test-time computation ( Snell et al. 2024 ) . Recent advancements, such as OpenAI-o1 ( OpenAI 2024 ) , Deepseek R1 ( DeepSeek-AI et al. 2025 ) , and Gemini 2.0 Flash Thinking ( DeepMind 2025 ) , have demonstrated that allowing LLMs to think before generating answers can significantly enhance performance and lead to the emergence of human-like reasoning patterns. These patterns like “Wait, hold on.” or “Let’s break this down.” , indicate that LLMs can develop a form of meta-thinking abilities that can generalize well to out-of-distribution (OOD) tasks ( Xiang et al. 2025 ) . Meta-thinking, also known as metacognitive skills ( Flavell 1979 ) , is an ability traditionally considered uniquely human ( Didolkar et al. 2024 ) . To cultivate meta-thinking patterns from LLMs themselves, recent construction-based supervised approaches leverage supervised finetuning on structured reasoning trajectories. Specifically, these methods sampling reasoning trajectories from predefined meta-thinking templates and then use supervised finetuning (SFT) or direct preference optimization (DPO) ( Rafailov et al. 2023 ) to teach LLMs imitate these patterns ( Qi et al. 2024 ; Yue et al. ; Xi et al. 2024 ; Yang et al. 2025 ; Muennighoff et al. 2025 ; Ye et al. 2025c ) . However, such methods lack sufficient flexibility for LLMs to explore suitable meta-thinking patterns. Thus, they often fail to generalize to out-of-distribution (OOD) problems, leading to unstable performance on unseen data ( Kirk et al. ; Chu et al. 2025 ) . Besides construction-based methods, R1-like single-agent reinforcement learning (SARL) has also been adopted for meta-thinking in reasoning ( DeepSeek-AI et al. 2025 ; Xie et al. 2025 ) . However, these SARL attempts typically rely on strong foundational models for easier exploration or extensive task-specific fine-tuning for stable training ( Xu et al. 2025 ; Gandhi et al. 2025 ) . Furthermore, SARL needs to learn meta-thinking and reasoning within a single forward pass, seeking to capture complex reasoning structures purely in an autoregressive manner ( Xie et al. 2025 ) . This can potentially lead to issues such as inefficient exploration as well as reduced readability and early convergence to local optima ( DeepSeek-AI et al. 2025 ; Xiang et al. 2025 ) . Figure 1 : Left: A construction-based method that fine-tunes LLMs using rejection sampling, searching among combinations of pre-defined templates. Middle: R1-like method learns to mix meta-thinking and detailed reasoning steps during training. Right: Our method ReMA separates the meta-thinking and reasoning steps in a multi-agent system and updated by reinforcement learning. To address these limitations, we introduce Re inforced M eta-thinking A gents (ReMA), a novel framework that leverages multi-agent reinforcement learning (MARL) to encourage LLMs to think about thinking . Our approach employs a multi-agent system (MAS) composed of a high-level meta-thinking agent, responsible for strategic oversight and instruction generation, and a low-level reasoning agent tasked with detailed executing processes based on provided guidance. We compare the inference process among the construction-based method, R1-like method, and ReMA in Fig. 1 . Since MAS distributes the exploration space of SARL into multiple agents, it enables each agent to explore more structurally and efficiently during training. Then we apply reinforcement learning on each agent with aligned reward functions. In this way, ReMA effectively balances the trade-off between generalization capability and exploration efficiency. As a result, they can learn to play the best of their role (either to meta-think or to follow instructions), at the present of the other agent. To our knowledge, we are the first to formally define and optimize a multi-agent meta-thinking reasoning process (MAMRP) through multi-agent reinforcement learning. Our extensive experiments span both math reasoning and LLM-as-a-Judge tasks, where ReMA consistently achieves the highest average performance across three backbone pretrained models. We further extend ReMA to multi-turn interaction settings on math reasoning tasks, implementing turn-level ratio to optimize trajectory returns and stabilize training. Through comprehensive ablation studies, we illustrate the evolving dynamics between agents, revealing unexpected interaction patterns such as role reversals under different reward settings. These findings provide valuable insights into how meta-thinking processes enhance the reasoning capabilities of LLMs. 2 Preliminaries In this section, we outline the formulation of the vanilla reasoning process (Sec. 2.1 ) and the representative training methods (Sec. 2.2 ) along with the notation used throughout the paper. 2.1 Vanilla Reasoning Process (VRP) The probability of generating a response \mathbf{y} equals the product of its stepwise probabilities. Given a model \pi_{\theta} and a prompt \mathbf{x}=(x_{1},\dots,x_{N}) , the vanilla reasoning process (VRP) autoregressively produces a response \mathbf{y}=(y_{1},\dots,y_{L}) with \pi_{\theta}(\mathbf{y}|\mathbf{x})=\prod\limits_{l=1}^{L}\pi_{\theta}(y_{l}|x_{1},x_{2},\ldots x_{N},y_{1},\ldots,y_{l-1})=\prod\limits_{l=1}^{L}\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x},\mathbf{y}_{<l}) The response usually contains intermediate reasoning steps before arriving at the final answer, this process is also known as chain-of-thought (CoT) ( Wei et al. 2022 ) , which can be represented as: \mathbf{x}\xrightarrow{\text{reasoning steps}}\mathbf{y}\sim\mathbf{a}, (1) where \mathbf{a} is the extracted final answer, which is included in the answer \mathbf{y} . 2.2 Training VRP via Reinforcement Learning RL frames VRP decoding process as a deterministic , token‐level Markov Decision process (MDP) ( Wang et al. 2024a ) . Its objective is \mathcal{J}(\theta)=\mathbb{E}_{(\mathbf{x},\mathbf{y}^{*})\sim\mathcal{D},\mathbf{y}\sim\pi_{\theta}}\left[R(\mathbf{y},\mathbf{y}^{*})\right]. where R(\cdot,\cdot) represents a reward function comparing generated answer \mathbf{y} with the golden answer \mathbf{y^{*}} for any question \mathbf{x} sampled from dataset \mathcal{D} . To compute the gradient \nabla_{\theta}\mathcal{J}(\theta) , computationally efficient algorithms GRPO ( Shao et al. 2024 ) and REINFORCE++ ( Hu 2025 ) are widely adopted. Take GRPO as an example, given a question-answer pair \mathbf{x},\mathbf{y}^{*} and a group of G generated responses \mathbf{y}_{i} , denote \mathbf{y}_{i,j}