LLM-Guided Reinforcement Learning: Addressing Training Bottlenecks through Policy Modulation

Heng Tan, Hua Yan, Yu Yang · 2025

Read LLM-Guided Reinforcement Learning: Addressing Training Bottlenecks through Policy Modulation by Heng Tan, Hua Yan, Yu Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20671.txt.

Introduction

Heng Tan and co-authors let a language model coach a learning agent out of ruts, in work from 2025. Reinforcement learning agents often settle for weak plans and miss long-term gains. Past fixes lean on costly retraining or on human notes that do not scale. This frame needs neither.

The language model reads the weak agent's past paths and flags the key states where choices went wrong. It then hints at better acts and quietly scores them to steer updates. Tests on standard game and control tasks beat top rivals. The authors credit plain word-based insight for breaking the training jam.

Catalog record

CREATORHeng Tan, Hua Yan, Yu Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20671

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLM-Guided Reinforcement Learning: Addressing Training Bottlenecks through Policy Modulation Heng Tan, Hua Yan, Yu Yang Abstract While reinforcement learning (RL) has achieved notable success in various domains, training effective policies for complex tasks remains challenging. Agents often converge to local optima and fail to maximize long-term rewards. Existing approaches to mitigate training bottlenecks typically fall into two categories: (i) Automated policy refinement, which identifies critical states from past trajectories to guide policy updates, but suffers from costly and uncertain model training; and (ii) Human-in-the-loop refinement, where human feedback is used to correct agent behavior, but this does not scale well to environments with large or continuous action spaces. In this work, we design a large language model-guided policy modulation framework that leverages LLMs to improve RL training without additional model training or human intervention. We first prompt an LLM to identify critical states from a sub-optimal agent’s trajectories. Based on these states, the LLM then provides action suggestions and assigns implicit rewards to guide policy refinement. Experiments across standard RL benchmarks demonstrate that our method outperforms state-of-the-art baselines, highlighting the effectiveness of LLM-based explanations in addressing RL training bottlenecks. 1 Introduction While Reinforcement Learning (RL) has shown strong potential in domains such as simulation games cai2023imitation ; liuutility , robotics yin2024offline ; shridhar2023perceiver , and traffic control systems tan2023joint ; huang2024uncertainty , training an optimal policy for complex tasks still remains challenging. RL training often encounters bottlenecks, such as getting stuck in local optima, which prevents the agent from achieving the maximum cumulative reward. Re-training policies from scratch to overcome these issues is often computationally expensive and time-consuming agarwal2022reincarnating ; vinyals2019grandmaster , which highlights the need for more efficient approaches to address training bottlenecks in RL. Existing approaches for addressing reinforcement learning (RL) training bottlenecks can be broadly divided into two categories. (i) Automated policy refinement with explanation: These methods begin with a non-optimal RL policy and train a separate network to identify critical states—states that strongly influence long-term rewards—as a form of explanation. Based on these identified states, different strategies are applied: recommending actions guo2021edge , preventing the agent from entering misleading state-action pairs liuutility , or encouraging exploration at those states cheng2023statemask ; cheng2024rice ; yu2023airs . However, these approaches face two key challenges: (1) definitions of critical states vary significantly across methods, and (2) training a network to reliably identify them introduces uncertainty and can be computationally expensive for complex tasks. (ii) Human-in-the-loop RL refinement: These methods rely on human experts to identify important time steps and either provide corrected actions or restrict certain actions at those points van2022correct ; guan2021widening . While effective in smaller settings, such approaches become less practical as the action space grows, due to increased cognitive load and scalability issues. Recent advances in reinforcement learning (RL) augmented by large language models (LLMs) have demonstrated the effectiveness of LLMs in natural language understanding, reasoning, and task planning, positioning them as a promising alternative to humans for guiding policy learning in complex tasks. For example, LLM can serve as reward designers, providing implicit or explicit rewards kwon2023reward ; wu2023read ; yu2023language , or as decision makers, offering suggested action candidates or prior policies li2022pre ; ahn2022can ; yao2020keep to guide RL policy learning. However, directly involving LLMs in the policy retraining process for complex tasks is prohibitively expensive in terms of both computational time and financial cost; for instance, in environments such as MuJoCo, the training process often requires over one million episodes on average cheng2023statemask . To address this challenge, our work explores how LLM-generated explanations can be used to mitigate RL training bottlenecks and enhance policy performance, without requiring full retraining. We introduce ULTRA, a framework of u tilizing large l anguage model-based explana t ion to address the training bottlenecks for r einforcement le a rning. Our method consists of two main components: (i) Identification: We first leverage an LLM to identify the critical states that are most influential to future rewards. Specifically, we collect historical trajectories from a non-optimal RL policy and provide them to the LLM, along with environment information and natural language-based identification instructions. The LLM processes this input to pinpoint the critical states within the trajectories. (ii) Refinement: Based on the identified critical states, we integrate the LLM into the agent’s Markov Decision Process (MDP) to refine its policy. We explore three strategies: (a) Given the identified critical states, we prompt the LLM to suggest optimal actions. These actions are stored in a lookup table, which the RL agent consults during policy fine-tuning. (b) We prompt the LLM to analyze the agent’s effective and sub-optimal actions at the critical states, producing explanations that guide reward shaping. (c) We combine both (a) and (b) to jointly leverage LLM-suggested actions and explanation-driven reward signals during the fine-tuning process. In summary, this paper makes the following contributions. • We introduce a framework that leverages LLM-based explanations to address training bottlenecks for RL. Specifically, we use an LLM to identify critical states from the historical trajectories of a non-optimal agent and explore three strategies for improving policy performance. • We evaluate our method across five environments, ranging from a simple sparse-reward game (e.g., Pong) to complex dense-reward tasks (e.g., MuJoCo [Ant]). Experimental results demonstrate that our approach outperforms state-of-the-art baselines. • We assess the explanation fidelity of different models and analyze how varying explanation affect the RL policy refinements. The experimental results show that our method achieves superior performance compared to existing baselines. 2 Related Work 2.1 Policy Refinement Reinforcement learning (RL) often suffers from inefficient training due to challenges such as sparse rewards, suboptimal exploration, and unstable policy updates. To address these bottlenecks, many existing methods focus on policy refinement—the process of improving a non-optimal policy by leveraging external signals or structural insights, which can be broadly classified into two categories: (i) automated policy refinement with explanation, and (ii) human-in-the-loop RL refinement. (i) Automated Policy Refinement with Explanation. Given a non-optimal RL policy, some approaches first discuss how the critical states are defined, and then train an auxiliary network to identify critical states as a form of explanation for the further policy refinement: perturbing the agent’s actions guo2021edge ; liuutility , or encouraging targeted exploration cheng2023statemask ; cheng2024rice ; yu2023airs . For example, StateMask cheng2023statemask learns to mask non-critical states while preserving the agent’s performance, revealing key decision points that impact the final reward. This insight supports downstream applications such as adversarial testing and error correction. UTILITY yu2023airs adopts a two-level explainability framework to detect agent mistakes and reformulate policy refinement as a bi-level optimization problem, incorporating reward shaping and constrained learning. However, these methods are sensitive to how critical states are defined, and the accuracy of the trained network significantly affects downstream performance. Moreover, training such networks for complex tasks is often computationally expensive and time-consuming. (ii) Human-in-the-Loop RL Refinement. Other methods incorporate expert feedback, where humans identify key time steps and either suggest improved actions or restrict undesirable ones van2022correct ; guan2021widening ; laleh2024survey ; metz2024mapping . For example, Van der Pol et al. van2022correct use human feedback after agent failures to create action-level shields that constrain high-level decisions, improving learning efficiency and policy robustness. Zhang et al.

Read the complete text