Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making
Xu Wan, Wenyue Xu, Chao Yang, Mingyang Sun · 2025
Read Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making by Xu Wan, Wenyue Xu, Chao Yang, Mingyang Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02522.txt.
Introduction
Xu Wan and three co-authors pair language models with reinforcement learning in a 2025 framework for giant decision tasks. Language models reason well but cannot steer long real-time chains. Learning agents steer well but waste samples when the move set is huge, as in power grid control with tens of thousands of moves.
In their loop, the language model plays two roles during training: it polishes the agent's poor moves, and it grades whole runs to assign credit. The agent, in turn, yields clean run data that fine-tunes the model for the task. Grid trials show the pair beats either side alone.
Catalog record
| CREATOR | Xu Wan, Wenyue Xu, Chao Yang, Mingyang Sun |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-02522
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making Xu Wan, Wenyue Xu, Chao Yang, Mingyang Sun Abstract Recent advancements in Large Language Models (LLMs) and Reinforcement Learning (RL) have shown significant promise in decision-making tasks. Nevertheless, for large-scale industrial decision problems, both approaches face distinct challenges: LLMs lack real-time long-sequence decision-making capabilities, while RL struggles with sample efficiency in vast action spaces. To bridge this gap, we propose A gents C o- E volution (ACE), a synergistic framework between LLMs and RL agents for large-scale decision-making scenarios. ACE introduces a dual-role trajectory refinement mechanism where LLMs act as both Policy Actor and Value Critic during RL’s training: the Actor refines suboptimal actions via multi-step reasoning and environment validation, while the Critic performs temporal credit assignment through trajectory-level reward shaping. Concurrently, RL agent enhances LLMs’ task-specific decision-making with high-quality fine-tuning datasets generated via prioritized experience replay. Through extensive experiments across multiple power grid operation challenges with action spaces exceeding 60K discrete actions, ACE demonstrates superior performance over existing RL methods and LLM-based methods. 1 Introduction Making effective control in large-scale physical systems has been a long-standing goal in artificial intelligence research ( Stoica et al. 2017 ) . Such decision-making tasks, including traffic control ( Zhang et al. 2022 ; Du et al. 2023a ) , power system operating ( Yoon et al. 2021 ; Dorfer et al. 2022 ; Chauhan et al. 2023 ) , and multi-robot coordination ( Mahler et al. 2019 ; Kalashnikov et al. 2022 ) , require sophisticated reasoning capabilities and rapid response mechanisms. Reinforcement Learning (RL) has been extensively studied as a promising approach to tackle these challenges for a long time. Through iterative interaction with the environment, RL agents learn optimal control policies by maximizing cumulative rewards. However, the paradigm of learning from scratch without prior knowledge ( Sutton & Barto 1999 ) making RL agents inherently suffer from sample inefficiency. Moreover, the time-varying and stochastic characteristics of large-scale industrial scenarios result in a substantial gap between the converged solutions obtained through RL models and true optimality ( Nian et al. 2020 ) . Various approaches leverage expert knowledge to alleviate the inefficiency and sub-optimality intrinsic in RL training. In terms of policy guidance, a straightforward yet effective approach is Learning from Demonstration (LFD) ( Argall et al. 2009 ) . LFD-based methods directly mimics expert demonstrations through behavior cloning or tries to optimize the reward function through Inverse Reinforcement Learning (IRL) ( Ng et al. 2000 ; Nair et al. 2018 ; Torabi et al. 2018 ; Adams et al. 2022 ) . While theoretically promising, LFD-based methods heavily rely on the quality of expert demonstrations and struggle to generalize to unseen scenarios, often suffering from distribution shift problems when encountering states not covered in the demonstration data. Meanwhile, Human-in-the-Loop (HITL) approaches ( Abel et al. 2017 ) has emerged as a promising paradigm that leverages expert feedback to adaptive guide policy learning. These methods incorporate human guidance either through real-time interaction ( Knox & Stone 2009 ; MacGlashan et al. 2017 ) or offline preference alignment ( Bai et al. 2022 ; Dai et al. 2023 ) , enabling human trainers to provide evaluative feedback or corrective advice during policy learning. However, due to the unpredictable and subjective nature of human feedback, HITL-based approaches face key challenges including intensive human time requirements, feedback inconsistency, and the balance between human intervention and autonomous learning ( Kumar et al. 2024 ) . Recent advances in Large Language Models (LLMs) have opened new possibilities for enhancing RL frameworks. Through training on massive text data, LLMs have acquired rich world knowledge and reasoning capabilities, making them promising candidates for decision-making tasks ( Lu et al. 2024 ; Wang et al. 2023 ) . Researchers have explored multiple LLM-RL integration paradigms: 1) semantic action space compression through LLM-guided abstraction ( Zhou et al. 2023 ) , 2) reward shaping via natural language instruction following ( Tan et al. 2024 ) , and 3) code generation for policy implementation ( Du et al. 2023b ) . Recent work like Thought Cloning ( Liu et al. 2024 ) further demonstrates how LLMs can generate chain-of-thought trajectories to guide policy learning through structured reasoning traces. However, these approaches face significant challenges in industrial control scenarios. While LLMs excel at high-level strategic planning, they struggle with long-sequence decision-making required for fine-grained control. Industrial tasks exhibit inherent time delays and temporal coupling, making it challenging for LLMs to generate coherent sequences of decisions over extended time horizons. Additionally, the autoregressive nature of transformer-based LLMs introduces significant latency ( Fu et al. 2024 ) , making them impractical for real-time control loops requiring sub-1000ms responses. Our analysis suggests that direct LLM-based action generation fundamentally conflicts with the precision and timing requirements of industrial control systems, necessitating a new paradigm that combines LLMs’ strategic reasoning with traditional RL’s numerical optimization strengths. Figure 1: The motivation and challenges of integrating LLM with RL for large-scale industrial decision making. Motivated by these observations, we develop a more suitable LLM-RL collaborative framework for industrial decision-making. Unlike previous works that integrate LLMs during inference, we restrict LLM as an offline guider in the training phase, preserving real-time performance during online interaction and deployment. Specifically, in the “ think twice ” phase, LLMs guide RL policy updates through two key mechanisms: (1) rectifying interaction trajectories as a Policy Actor to refine suboptimal decisions, and (2) performing trajectory-level reward shaping as a Value Critic to enable better credit assignment. In the “ act once ” phase, RL agent interacts with the environment to generate RL trajectories, which are then combined with LLM-refined trajectories to create a mixed experience buffer. This buffer serves both as a training buffer for RL and as a fine-tuning dataset to enhance LLM’s task-specific guidance capabilities. To summarize, our main contributions are: (1) We propose A gents C o- E volution (ACE), a co-evolution framework that separates LLM reasoning and RL execution into offline training and online deployment, enabling both effective learning and real-time decision-making in large-scale industrial scenarios. (2) For guiding efficient exploration in large-scale decision-making, we develop a dual-role trajectory refinement mechanism where LLMs serve as both Policy Actor and Value Critic, addressing sample inefficiency and solution sub-optimality, respectively. (3) For enabling continuous improvement of RL and LLMs, we establish an automated high-quality dataset generation workflow through reward-based prioritization and weighted policy update strategies in experience gathering. (4) We demonstrate state-of-the-art performance on 3 L2RPN competitions, outperforming existing expert-guided RL methods and LLM methods in industrial environments with over 60K action space. 2 Related Work Expert-Guided Reinforcement Learning Expert guidance has been widely explored to address the sample efficiency and exploration challenges inherent in RL. Imitation learning-based methods, like DAgger ( Ross et al. 2011 ) , Hg-DAgger ( Le et al. 2018 ) , Soft DAgger ( Nazeer et al. 2023 ) iteratively collect expert feedback on states visited by the learning policy to address distribution shift. Instead of direct action imitation, IRL-based methods select a reward function from the set of possible solutions that best explains expert behavior. AIRL ( Fu et al. 2017 )