LaGO: Latent Action Guidance for Online Reinforcement Learning

Kuan-Yen Liu, Ren-Jyun Huang, Ti-Rong Wu · 2026

Read LaGO: Latent Action Guidance for Online Reinforcement Learning by Kuan-Yen Liu, Ren-Jyun Huang, Ti-Rong Wu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24669.txt.

Introduction

Liu, Huang, and Wu let a language model whisper guidance to a reinforcement learner instead of ordering it around. Language models plan well but make unreliable direct controllers, since their actions must be generated precisely. LaGO instead uses a pretrained model as a latent action prior, softly steering online policy optimisation while the reinforcement learner keeps final control of every action.

On a discrete robot benchmark and a continuous-control suite, LaGO consistently improves both reward and success rate over vanilla proximal policy optimisation, the standard baseline. Success rates multiply severalfold on the continuous benchmark and nearly double on the discrete one. Stronger pretrained language models provide more effective guidance, suggesting the benefit is genuine transferred knowledge rather than noise. Liu's 2026 design occupies a middle ground between trusting the language model outright and ignoring it: its knowledge shapes the search, while interaction with the environment still arbitrates what survives.

Catalog record

CREATORKuan-Yen Liu, Ren-Jyun Huang, Ti-Rong Wu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24669

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LaGO: Latent Action Guidance for Online Reinforcement Learning Kuan-Yen Liu, Ren-Jyun Huang, Ti-Rong Wu Abstract Large language models (LLMs) have shown strong potential for planning and sequential decision-making, but prior work often relies on using them as direct controllers, which requires precise action generation and can be unreliable in practice. This paper proposes Latent Action Guidance for Online Reinforcement Learning (LaGO) , a framework that uses a pretrained LLM as a latent action prior to softly guide online policy optimization, rather than treating the LLM as an explicit planner or controller. Experiments on both a discrete-control benchmark, CLEVR-Robot, and a continuous-control benchmark, Meta-World, demonstrate that LaGO consistently improves both reward and success rate over Vanilla PPO. In particular, LaGO increases the average success rate from 15.1% to 27.2% on CLEVR-Robot and from 2.7% to 15.2% on Meta-World. Our analysis further shows that stronger pretrained LLMs provide more effective guidance, suggesting that LLM knowledge can improve planning and online decision-making. † † affiliationnotice: Equal contribution 1 Introduction Large language models (LLMs) have shown remarkable capabilities across a wide range of tasks ( 18 ; 22 ; 9 ; 26 ) , suggesting that pretrained models acquire rich knowledge about environments, tasks, and possible behaviors. Motivated by this, recent studies have explored the use of LLMs in reinforcement learning (RL) and sequential decision-making problems. For example, some works use LLMs as direct controllers ( 2 ; 20 ; 27 ) , where the model predicts actions from observations or multimodal inputs. Other works use LLMs as high-level planners ( 1 ; 13 ; 23 ; 12 ) , where the model proposes subgoals or task plans and a separate low-level RL policy executes them. Furthermore, LLMs have also been used as internal world model simulators ( 11 ; 17 ; 3 ; 21 ; 7 ) , where the model generates imagined future trajectories to predict environment dynamics for planning. These results highlight the potential of LLMs as a useful component for planning and RL. Despite these promising results, existing approaches often require LLMs to provide sufficiently accurate outputs, such as actions, plans, or imagined trajectories, in order to be useful for RL control. However, this requirement can be overly strong in practice, since the capabilities of LLMs may vary substantially across different models. As a result, directly relying on an LLM as the primary controller may be unreliable, especially in RL problems that require precise execution and long-horizon reasoning. In such settings, even small prediction errors may accumulate over time and lead to poor control performance. Therefore, although LLMs may contain useful prior knowledge, using them for explicit control in RL remains challenging. To address this challenge, this paper investigates whether LLMs can be used not as explicit controllers but as a source of soft guidance for RL. Specifically, we propose Latent Action Guidance for Online Reinforcement Learning (LaGO) , a two-stage framework for leveraging LLM knowledge in online RL. In the first stage, LaGO uses expert demonstrations to fine-tune a pretrained LLM to learn a latent action guidance model. Then, the learned model provides action guidance signals as inductive priors for policy learning during online RL training. This substantially relaxes the accuracy requirement of using LLMs as direct controllers, since the LLM only needs to provide a coarse but informative guidance signal rather than exact decisions. Experiments on both discrete control tasks on CLEVR-Robot and continuous control tasks in Meta-World demonstrate that LaGO consistently improves both reward and success rate over vanilla PPO. Specifically, it increases the average success rate from 15.1% to 27.2% in CLEVR-Robot, and from 2.7% to 15.2% in Meta-World. The results demonstrate that LLMs can still help RL training even when they are not sufficiently accurate to serve as direct controllers. Furthermore, our results show that the quality of the pretrained LLM models has a substantial impact on the effectiveness of the proposed method. These findings suggest that stronger foundation LLMs are more likely to provide useful guidance signals for RL training in the future. Overall, LaGO provides a simple and practical framework for leveraging pretrained LLM knowledge as latent action guidance for planning and RL. 2 Related Work 2.1 LLMs as Direct or Hierarchical Controllers Recent progress in Vision-Language-Action (VLA) models ( 2 ; 6 ; 15 ) has demonstrated that large-scale pretraining foundation LLM models can be adapted to directly output control signals in embodied environments. These methods demonstrate the potential of using foundation models as direct controllers, especially when rich visual and language inputs are available. However, using LLMs as end-to-end controllers remains challenging in reinforcement learning settings, as it requires precise action execution, particularly in vectorized environments where observations and actions are not naturally represented in language or visual form. An alternative approach is to adopt a hierarchical paradigm ( 1 ; 23 ; 13 ) , where the LLM serves as a high-level planner that proposes textual subgoals or instructions, while a separate low-level policy handles execution. This design avoids requiring the LLM to perform direct low-level control, but still relies on explicit textual outputs, and effectively integrating these outputs into RL policy learning remains a challenge. 2.2 LLMs as World Models for Planning LLMs have been shown to encode structured world knowledge about environments, including task dynamics, physical regularities, and action consequences, acquired during large-scale pretraining ( 16 ; 4 ) . Recent studies have further provided evidence that world-relevant information is preserved in LLM hidden representations. For example, 10 demonstrate that spatial and temporal information can be linearly decoded from LLM hidden states, while 14 show that sequence models can encode future environment configurations in their latent state. Motivated by this, several works have attempted to leverage such world knowledge from LLMs for planning and sequential decision-making. 17 incorporate language into a world model to improve future prediction and policy learning. 24 improve language models by finetuning them on embodied experiences collected from simulators. Despite these advances, these approaches still require LLMs to explicitly and accurately simulate valid future states or trajectories. To address this, recent work such as KALM ( 19 ) first fine-tunes an LLM on expert demonstrations, and then uses the fine-tuned model to generate additional rollouts under offline RL settings. While effective, this approach does not directly extend to the online setting. In contrast, our work investigates whether LLMs can also improve reinforcement learning in the online setting. 2.3 LLMs as Priors for Reinforcement Learning In reinforcement learning, another natural idea is to introduce a prior to guide policy learning and exploration. Recent work has begun to explore LLMs in this direction. For example, 25 treat LLMs as the prior action distribution and incorporate them into both policy-based and value-based RL frameworks from a Bayesian inference perspective, demonstrating that fixed LLM priors can improve learning efficiency. However, this and related approaches ( 27 ; 5 ) are still limited to text-based or highly discretized action spaces. In conclusion, compared to previous works, our work investigates a different way of leveraging LLMs in reinforcement learning. Rather than using LLMs as direct controllers, explicit world model simulators, or text-based action priors, we study whether their knowledge can benefit online RL through latent representations rather than explicit text.

Read the complete text